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Executive  Summary 


ES.1  Introduction 

This  report  is  intended  to  preserve  the  Joint  Advanced  Distributed  Simulation  (JADS) 
management  experience  for  future  test  planners  and  directors:  what  did  and  what  did  not  work; 
management  lessons  learned;  assessments  of  JADS  success  in  achieving  chartered  goals;  and 
conclusions  and  recommendations  on  the  JADS  management  experience. 

ES.2  Program  Overview 

JADS  was  chartered  in  October  1994  to  investigate  the  utility  of  advanced  distributed  simulation 
(ADS)  for  both  developmental  test  and  evaluation  (DT&E)  and  operational  test  and  evaluation 
(OT&E).  JADS  investigated  the  present  utility  of  ADS,  including  distributed  interactive 
simulation,  for  test  and  evaluation  (T&E);  identified  the  critical  constraints,  concerns,  and 
methodologies  when  using  ADS  for  T&E;  and  finally,  identified  the  requirements  that  must  be 
introduced  into  ADS  programs  if  they  are  to  support  a  more  complete  T&E  capability  in  the 
future. 

In  order  to  provide  the  T&E  community  with  tangible  proof  of  the  utility  of  ADS  as  a 
methodology,  JADS  performed  three  tests:  the  System  Integration  Test  (SIT)  explored  ADS 
support  of  precision  guided  munitions  (PGM)  testing,  the  End-To-End  (ETE)  Test  investigated 
ADS  support  for  command,  control,  communications,  computers,  intelligence,  surveillance  and 
reconnaissance  (C4ISR)  testing,  and  the  Electronic  Warfare  (EW)  Test  examined  ADS  support 
for  EW  testing.  The  joint  test  force  was  also  chartered  to  observe,  or  participate  at  a  modest  level 
in,  ADS  activities  sponsored  and  conducted  by  other  agencies  in  an  effort  to  broaden  conclusions 
developed  in  the  three  dedicated  test  areas. 

The  following  is  a  summary  of  the  three  JADS  test  programs. 

The  System  Integration  Test  investigated  the  utility  of  ADS  to  complement  T&E  of  precision 
guided  munitions.  SIT  was  a  two-phase  test.  Phase  1,  the  Linked  Simulators  Phase  linked 
hardware-in-the-loop  laboratories  (HWIL)  at  the  Naval  Air  Warfare  Center  Weapons  Division 
(Point  Mugu,  California,  and  China  Lake,  California)  representing  the  shooter  and  target  with  an 
air  intercept  missile  (AIM)-9  Sidewinder  HWIL  to  execute  a  closed-loop,  air-to-air  engagement. 
Phase  2,  the  Live  Fly  Phase  (LFP)  was  conducted  at  Eglin  Air  Force  Base,  Florida,  where  live 
aircraft  flying  over  the  Gulf  Test  Range  represented  the  shooter  and  target  and  were  linked  to  an 
AIM- 120  advanced  medium  range  air-to-air  missile  HWIL  simulation.  LFP  had  both  open  and 
closed  loops. 

The  End-to-End  Test  examined  the  utility  of  ADS  to  complement  the  T&E  of  a  C4ISR  system. 
ADS  was  used  to  provide  a  robust  test  environment  with  a  representative  number  of  threats  and 
complementary  suite  of  friendly  C4ISR  and  weapon  systems  with  which  the  system  under  test 
interacted.  The  Joint  Surveillance  Target  Attack  Radar  System  (Joint  STARS)  suite  of  E-8C 
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aircraft  and  ground  station  module  was  chosen  as  a  representative  C4ISR  system.  The  ETE  Test 
was  a  four-phase  test.  The  first  two  phases  occurred  in  a  laboratory  environment  suited  for 
exploring  DT&E  and  early  OT&E  applications.  Phase  3  checked  compatibility  of  the  ADS 
environment  with  the  actual  Joint  STARS  equipment,  and  Phase  4  augmented  live  open  air  tests 
with  a  virtual  battlefield  in  real  time  evaluating  operational  measures  of  performance. 

The  Electronic  Warfare  Test  combined  the  results  of  three  leveraged  activities  with  a  three-phase 
test  of  a  self-protection  jammer  (SPJ).  The  leveraged  activities  were  the  Defense  Modeling  and 
Simulation  Organization’s  (DMSO)  High  Level  Architecture  (HLA)  Engineering 
Protofederation,  Office  of  the  Secretary  of  Defense’s  CROSSBOW  Threat  Simulator  Linking 
Activity  (TSLA)  Study,  and  the  U.S.  Army’s  Advanced  Distributed  Electronic  Warfare  System 
(ADEWS).  The  first  phase  of  the  SPJ  test  was  a  baseline  data  collection  phase  executed  at  both 
an  open  air  range  and  a  hardware-in-the-loop  facility.  Phases  2  and  3  replicated  Phase  1  in  an 
ADS  environment  linking  the  JADS  Test  Control  and  Analysis  Center,  Albuquerque,  New 
Mexico,  with  the  Air  Force  Electronic  Warfare  Environment  Simulator  (AFEWES)  in  Fortworth, 
Texas  and  the  Navy’s  Air  Combat  Environment  Test  and  Evaluation  Facility  (ACETEF)  in 
Patuxent  River,  Maryland.  The  SPJ  was  represented  with  a  digital  computer  model  in  Phase  2, 
while  an  actual  SPJ  in  an  installed  system  test  facility  was  used  in  Phase  3. 

ES.3  Assessment  of  Accomplishments 

Two  assessments  were  conducted:  accomplishment  of  the  mission  in  the  JADS  charter  and 
accomplishment  of  the  JADS  legacy. 

JADS  did  not  execute  within  its  original  schedule  or  budget.  Of  the  three  chartered  tests,  the  SIT 
required  restructuring  because  of  a  12-month  delay  in  getting  a  key  piece  of  software  developed 
by  a  missile  program  office.  The  ETE  Test  execution  had  to  be  delayed  12  months  as  well 
because  of  inadequate  funding  in  the  first  year  of  the  test  and  difficulty  getting  a  contract  vehicle 
to  Northrop  Grumman  set  up.  These  delays  increased  the  requirement  for  infrastructure  and 
contractor  support  in  the  final  year  of  JADS.  Additionally,  a  mid-test  assessment  of  technical 
risk  determined  that  JADS  had  unacceptable  funding  margins  based  on  the  risk  involved  in  the 
remaining  test  activities.  As  a  result,  the  joint  test  director  (JTD)  provided  a  decision  briefing 
requesting  additional  funding  to  the  Deputy  Director  of  Systems  Assessment  who  provided 
another  $3.1  million  in  May  1997. 

In  spite  of  the  schedule  and  budget  changes,  it  is  the  JTD’s  assessment  that  JADS  was  nearly  95 
percent  successful  in  accomplishing  its  chartered  mission.  The  results  of  the  tests  were  the 
primary  basis  for  addressing  the  three  tasks  in  the  charter.  Present  utility  was  determined  to  be 
case  specific  and  proven  to  exist  for  certain  classes  of  systems  and  suspect  for  others;  concerns, 
constraints,  and  methodologies  as  well  as  requirements  to  facilitate  increased  use  of  ADS  were 
identified  and  published  in  test  and  special  reports.  Inroads  were  made  into  many  test 
organizations  and  acquisition  programs  that  increased  the  use  of  ADS  in  all  facets  of  test  and 
evaluation.  Significant  positive  impact  was  made  by  JADS  on  the  development  of  the 
Department  of  Defense’s  high  level  architecture  ensuring  the  needs  of  the  T&E  community  were 
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addressed.  Although  there  were  many  challenges,  the  work  was  technically  stimulating  and  the 
joint  test  force  (JTF)  was  highly  motivated  to  succeed. 

Overall,  the  JADS  legacy  program  was  an  unqualified  success.  Newsletter,  conference 
participation,  web  site,  “ADS  for  T&E”  professional  development  course,  lessons  learned  videos 
and  multimedia  report  formats  allowed  JADS  to  reach  and  influence  a  broad  swath  of  the  T&E 
community,  both  government  and  civilian.  The  T&E  community  was  educated  to  at  least 
consider  using  ADS  to  support  future  test  programs.  Testers  were  equipped  with  the  basics 
needed  to  successfully  plan  and  execute  an  ADS  test,  and  JADS’  products  were  institutionalized 
to  provide  a  firm  foundation  on  which  to  build.  Success  did  not  come  easily  in  legacy.  The 
tendency  of  the  JTF  was  to  focus  on  the  successful  execution  of  the  three  tests.  Developing  a 
legacy  plan  early  in  the  program  and  assigning  a  high  priority  to  legacy  (including  a  full-time 
legacy  manager)  as  well  as  committing  much  of  the  service  deputies’  time  to  legacy  activities 
were  critical  to  this  effort.  Subsequent  “nagging”  was  required  by  the  JADS  staff  to  ensure 
legacy  actions  were  occurring  early  enough  and  long  enough  to  have  an  impact  on  the  T&E 
community.  Finally,  many  trips  were  required  to  get  the  users’  attention  through  face-to-face 
meetings. 

ES.4  Lessons  Learned,  Conclusions  and  Recommendations 

JADS  has  enjoyed  a  reputation  of  being  a  particularly  successful  JT&E  program.  Although  this 
will  come  across  as  over  simplified,  JADS’  success  can  be  attributed  in  a  simple  formula  - 
excellent  organization  plus  proactive  legacy  actions  equals  successful  JT&E. 

JADS  used  an  approach  to  JTF  organization  that  is  quite  different  from  what  the  JT&E 
Handbook  calls  for.  The  point  is  this  -  the  Handbook  is  a  guide.  However,  each  JT&E  program 
is  going  to  be  unique.  Therefore,  structure  the  organization  in  a  way  that  best  makes  sense  for 
your  particular  mission  and  working  environment.  Having  said  that,  there  are  facets  of  JADS’ 
approach  we  recommend  to  you  regardless  of  the  type  of  JT&E  you  have. 

•  The  mix  of  matrix  and  functional  area  teams  within  JADS  worked  exceptionally  well. 
On  a  related  note,  having  all  government  and  contractor  personnel  together  and 
integrated  into  teams  made  for  a  highly  cohesive  team  that  worked  together  for  the 
common  good. 

•  Having  the  JTF  under  one  single  roof  made  mission  execution  an  order  of  magnitude 
easier  than  if  we  had  been  geographically  separated. 

•  Possessing  all  support  functions  within  the  JTF  is,  in  our  opinion,  the  only  way  to  go. 
We  witnessed  innumerable  cases  of  “non-support”  from  other  agencies.  JADS  was 
always  able  to  get  the  support  we  needed  because  we  owned  the  support  functions.  In 
fact  we  found  ourselves  supporting  other  JT&Es  as  well. 

•  We  started  with  a  flat  organization,  but  inserted  a  Chief  of  Staff  when  it  became 
apparent  the  JTD  traveled  too  much  to  provide  day-to-day,  hands-on  leadership.  This 
worked  well. 

JADS  has  been  both  criticized  and  praised  for  its  legacy  program.  On  one  hand,  JADS  was 
praised  for  having  the  best  legacy  program  ever  seen.  Though  biased,  we  agree.  Criticism,  on 
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the  other  hand,  fell  in  two  areas  -  it  cost  too  much  and  isn’t  relevant  to  other  types  of  JT&Es. 
JADS’  experience  is  that  you  can  have  a  highly  comprehensive  legacy  program  for  a  modest 
investment.  We  devoted  approximately  four  percent  of  our  workforce  (2  of  50)  and  1.5  percent 
of  our  budget  for  legacy.  As  for  relevance,  “experts”  said  that,  since  JADS  was  a  different  type 
of  JT&E  than  most,  other  JT&Es  shouldn’t  need  to  do  this  aggressive  style  of  approach  to  legacy. 
We  beg  to  differ.  Regardless  of  the  nature  of  your  JT&E,  you  have  a  user  community  you’re 
working  with,  and  both  information  and  products  you  need  to  embed  into  this  community  to 
make  lasting  positive  contributions.  Therefore,  the  basic  precepts  of  JADS’  legacy  program  hold 
true,  only  the  details  differ.  Here  are  some  key  points: 

•  Make  a  commitment  to  a  legacy  program  from  the  very  beginning  of  your  JT&E 
program.  Build  it  into  your  organizational  structure,  man  it  and  fund  it  from  Day  1 . 

•  A  successful  legacy  program  needs  the  full  support  of  the  JT&E  leadership,  most 
especially  the  JTD  and  Deputies. 

•  Look  for  every  way  possible  to  get  your  word  out  on  a  continuing  basis  through  the 
life  cycle  of  your  program.  Interim  reports,  newsletters,  videos,  CDs,  presentations  at 
conferences  your  community  attends,  etc.  should  all  be  used. 

•  JADS  was  highly  successful  across  the  board.  As  an  organization,  it  was  praised  for  its 
cohesiveness,  high  morale,  and  expertise  in  getting  the  job  done.  As  examples,  our 
personnel,  computer  and  finance  people  were  called  upon  to  help  many  others.  JADS  was 
also  very  successful  in  conducting  rigorous  T&E  events  which  fulfilled  JADS  charter  and 
provided  valid,  believable  data  to  our  user  communities.  Finally,  JADS  is  making 
widespread  and  long-lasting  contributions  to  the  community  thanks  to  its  superb  legacy 
program. 
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1.0  Introduction 


This  report  is  intended  to  preserve  the  Joint  Advanced  Distributed  Simulation  (IADS) 
management  experience  for  future  test  planners  and  directors:  what  did  and  what  did  not  work; 
management  lessons  learned;  assessments  of  IADS  success  in  achieving  chartered  goals;  and 
conclusions  and  recommendations  on  the  IADS  management  experience. 

2.0  Program  Overview 

The  JADS  Joint  Test  and  Evaluation  (JT&E)  was  chartered  by  the  Deputy  Director,  Test, 
Systems  Engineering  and  Evaluation  (Test  and  Evaluation),  Office  of  the  Under  Secretary  of 
Defense  (Acquisition  and  Technology)  in  October  1994  to  investigate  the  utility  of  advanced 
distributed  simulation  (ADS)  technologies  for  support  of  developmental  test  and  evaluation 
(DT&E)  and  operational  test  and  evaluation  (OT&E).  The  program  was  Air  Force  led  with 
Army  and  Navy  participation.  The  joint  test  force  (JTF)  manning  included  23  Air  Force,  13 
Army,  and  2  Navy  personnel.  Science  Applications  International  Corporation  (SAIC)  and 
Georgia  Tech  Research  Institute  (GTRI)  provided  technical  support  that  varied  between  15  and 
20  man-years  per  year.  The  program  was  completed  within  its  five-year  schedule. 

The  original  JADS  JT&E  charter  included  only  the  End-to-End  (ETE)  Test  and  the  System 
Integration  Test  (SIT),  but  the  Deputy  Director,  Test,  Systems  Engineering  and  Evaluation  tasked 
the  JADS  JT&E  to  conduct  a  second  feasibility  study  expanding  the  test  to  include  an  electronic 
warfare  (EW)  environment.  This  feasibility  study  was  conducted  concurrent  to  the  start  up  of  the 
JADS  JT&E  program  with  no  additional  manpower  added  to  the  organization.  The  feasibility 
results  were  presented  to  the  senior  advisory  council  (SAC)  in  1995  where  it  was  recommended 
that  JADS  revise  the  proposed  EW  Test  to  lower  costs.  JADS  revised  the  test  and  presented  the 
results  to  the  SAC  in  1996.  At  that  time,  the  SAC  approved  the  addition  of  the  EW  Test  to  the 
JADS  mission. 

A  high-level  overview  schedule  of  the  JADS  JT&E  program  is  shown  in  Table  1. 

Table  1.  JADS  Schedule 


Date 

Activity 

Oct  93 

Joint  Feasibility  Study  (JFS)  Charter 

Oct  94 

JT&E  Charter  (minus  EW  Test) 

Aug  96 

EW  Test  Charter 

Nov  97 

SIT  Complete 

Aug  99 

ETE  Test  Complete 

Nov  99 

EW  Test  Complete 

Mar  00 

JADS  Complete 
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The  JADS  problem  domain  included  developmental  testing  (DT)  and  operational  testing  (OT)  of 
all  types  of  weapon  systems.  Obviously,  the  JADS  JT&E  could  not  conduct  a  DT  and  OT  test 
for  every  kind  of  weapon  system  possible.  Therefore,  the  JADS  JT&E  selected  as  many 
applications  as  time  and  resources  permitted. 

As  finally  approved,  the  JADS  JT&E  program  included  three  tests:  SIT,  ETE  Test  and  EW  Test. 
The  following  is  a  summary  of  the  three  test  programs. 

System  Integration  Test.  The  SIT  investigated  the  utility  of  ADS  in  complementing  test  and 
evaluation  (T&E)  of  precision  guided  munitions  (PGMs).  The  air  intercept  missile  (AIM)-9 
Sidewinder  and  AIM- 120  advanced  medium  range  air-to-air  missile  (AMRAAM)  were  chosen. 
Both  DT&E  and  OT&E  aspects  were  explored.  DT&E  applications  were  explored  using  a 
hardware-in-the  loop  facility  to  simulate  the  missile.  This  allowed  detailed  performance  of 
missile  subsystems  to  be  monitored,  typical  of  DT&E.  The  OT&E  characteristics  of  the  SIT 
result  from  the  use  of  actual  aircraft  performing  operationally  realistic  engagements.  Of 
particular  value  was  the  launched  aircraft  fire  control  radar  which  operated  in  the  real 
environment  and  was  affected  by  weather,  electronic  countermeasures,  clutter,  and  other 
variables  for  which  good  digital  models  do  not  exist.  This  meant  that  the  T&E  was  more 
representative  of  the  performance  of  the  integrated  weapon  systems.  SIT  was  a  two-phase  test. 
Phase  1  activities  were  conducted  at  Weapons  Division  (NAWC-WPNS)  (Point  Mugu, 
California,  and  China  Lake,  California),  and  Phase  2  activities  were  conducted  at  Eglin  Air  Force 
Base  (APB),  Florida. 

End-to-End  Test.  The  ETE  Test  examined  the  utility  of  ADS  to  complement  the  DT&E  and 
OT&E  of  a  command,  control,  communications,  computers,  intelligence,  surveillance  and 
reconnaissance  (C4ISR)  system.  ADS  was  used  to  provide  a  more  robust  test  environment  that 
provided  more  representative  numbers  of  threats  plus  the  complementary  suite  of  other 
command,  control,  communications,  computers  and  intelligence  (C4I)  and  weapon  systems  with 
which  the  system  under  test  interacted.  The  Joint  Surveillance  Target  Attack  Radar  System 
(Joint  STARS)  suite  of  E-8C  aircraft  and  ground  station  module  was  chosen  as  a  representative 
C4I  system  on  which  to  introduce  ADS  as  a  methodology  in  both  DT&E  and  OT&E  settings. 
The  ETE  Test  was  a  four-phase  test.  The  first  two  phases  occurred  in  a  laboratory  environment 
suited  for  exploring  DT&E  and  early  OT&E  applications.  Phase  3  checked  compatibility  of  the 
ADS  environment  with  the  actual  Joint  STARS  equipment,  and  Phase  4  combined  live  open  air 
tests  with  laboratory  tests  evaluating  operational  measures  of  performance  in  a  notional  corps 
scenario. 

Electronic  Warfare.  The  EW  Test  combined  the  results  of  three  leveraged  activities  with  a 
three-phase  test  of  a  self-protection  jammer  (SPJ).  This  multivectored  approach  was  designed  to 
assess  the  utility  of  ADS  to  EW  T&E  by  testing  the  ability  of  ADS  technology  to  provide 
improved  performance  for  EW  T&E  within  an  acceptable  cost  and  schedule.  The  leveraged 
activities  were  the  Defense  Modeling  and  Simulation  Organization’s  (DMSO)  High  Level 
Architecture  (HLA)  Engineering  Protofederation,  Office  of  the  Secretary  of  Defense’s  (OSD) 
CROSSBOW  Threat  Simulator  Linking  Activity  (TSLA)  Study,  and  the  U.S.  Army’s  Advanced 
Distributed  Electronic  Warfare  System  (ADEWS).  The  leveraged  activities  provided  qualitative 


6 


data  to  be  combined  with  the  SPJ  test,  which  provided  quantitative  data  to  assess  the  ability  of 
ADS  to  solve  the  inherent  limitations  of  the  EW  test  process.  These  data  were  also  used  to 
evaluate  the  potential  enhancements  to  the  EW  test  process  available  through  ADS. 

The  approach  was  to  take  historical  data  from  previously  executed  tests,  replicate  those  test 
environments  in  an  ADS  architecture  and  compare  the  results.  These  baseline  comparisons  were 
used  to  establish  the  validity  of  the  data  provided  by  ADS  testing  for  each  of  the  test  programs. 
Once  this  baselining  was  accomplished,  an  assessment  was  made  of  where  ADS  could  be  used  to 
address  shortfalls  in  conventional  testing.  An  assessment  of  the  critical  ADS  implementation 
issues  was  also  made. 

During  the  life  of  JADS,  there  were  many  non-JADS  ADS  tests  or  demonstrations  conducted. 
The  JADS  JTF  participated  with  many  of  these  activities  and  surveyed  many  others  that 
complimented  the  three  JADS  test  programs.  The  results  from  these  non-JADS,  ADS-enhanced 
tests  were  used  to  supplement  the  JADS  specific  results. 

Once  the  results  from  specific  systems  were  obtained  and  analyzed,  the  JADS  JTF  extended  or 
extrapolated  these  results  to  classes  of  systems.  Classes  of  systems  were  defined  by  example  as 
air-to-air  missiles,  aircraft,  C4I  systems,  EW  systems,  submarines,  spacecraft,  etc.  Each  of  these 
classes  of  systems  presented  unique  challenges  to  the  T&E  community  responsible  for  its 
evaluation.  The  final  step  was  to  take  the  results  of  those  classes  of  systems  for  which  ADS  test 
data  were  available  and  to  extend  them  to  as  much  of  the  total  JADS  problem  as  possible. 

The  test  issues  for  JADS  JT&E  are  shown  in  Table  2. 

Table  2.  JADS  Test  Issues 


Test  Issue  #1 

What  is  the  present  utility  of  ADS,  including  distributed  interactive  simulation 
(DIS),forT&E? 

Test  Issue  #2 

What  are  the  critical  constraints,  concerns,  and  methodologies  when  using  ADS 
for  T&E? 

Test  Issue  #3 

What  are  the  requirements  that  must  be  introduced  into  ADS  systems  if  they  are 
to  support  a  more  complete  T&E  capability  in  the  future? 
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3.0  Assessment  of  Accomplishment  of  JADS  Mission 

This  section  contains  two  assessments.  The  first  is  an  assessment  of  the  accomplishment  of  the 
mission  in  the  JADS  charter,  and  the  second  is  an  assessment  of  transfer  of  JADS  information 
and  products  to  the  services  and  OSD  which  will  be  referred  to  as  the  accomplishment  of  the 
JADS  legacy. 

3.1  Accomplishment  of  JADS  Charter 

The  following  is  the  critical  tasking  from  the  JADS  charter. 

“JADS  is  chartered  to  investigate  the  utility  of  advanced  distributed  simulation  (ADS)  for  both 
developmental  test  and  evaluation  (DT&E)  and  operational  test  and  evaluation  (OT&E).  JADS 
will  investigate  the  present  utility  of  ADS,  including  distributed  interactive  simulation  (DIS),  for 
T(&E;  identify  the  critical  constraints,  concerns,  and  methodologies  when  using  ADS  for  T&E; 
and  finally,  identify  the  requirements  that  must  be  introduced  into  ADS  systems  if  they  are  to 
support  a  more  complete  T&E  capability  in  the  future.” 

Utility  was  assessed  by  first  determining  if  the  ADS-supported  tests  produced  valid  test  data. 
This  was  done  by  comparing  ADS-produced  test  data  with  previously  conducted  DT&E  or 
OT&E  test  data.  While  validity  was  considered  to  be  an  essential  condition  for  utility,  it  was  not 
considered  to  be  a  sufficient  condition.  For  ADS  to  have  utility  it  also  had  to  have  benefits  over 
conventional  test  methods.  Benefits  included  cost  savings  as  well  as  being  able  to  overcome 
conventional  test  limitations.  The  SIT,  ETE  Test,  and  EW  Test  phase  reports  documented 
validity  and  benefits  for  the  representative  systems  used  in  those  tests.  PGM,  C4ISR,  and  EW 
class  utility  was  then  assessed  in  class  reports  by  combining  the  three  test  results  respectively 
with  other  ADS  results.  The  JADS  Final  Report  combined  these  results  and  assessments  to  make 
a  general  assessment.  Although  the  original  approach  of  determining  validity  by  comparing  ADS 
data  with  previously  conducted  conventional  DT&E  or  OT&E  test  proved  difficult,  we  were 
able,  in  most  cases,  to  develop  alternate  methods  which  satisfactorily  demonstrated  the  validity 
of  the  ADS-generated  data.  The  benefit  of  overcoming  conventional  test  limitations  was 
successfully  addressed  by  determining  which  of  40  previously  identified  common  test  limitations 
could  be  overcome  by  the  three  JADS  tests.  The  ability  to  demonstrate  cost  savings  was  only 
partially  successful.  Although  several  cost  comparison  studies  that  showed  cost  savings  for 
specific  programs  were  conducted,  the  results  were  very  assumption  dependent.  A  general  cost 
comparison  methodology  was  developed  and  documented  in  the  JADS  Special  Report  on  the 
Cost  and  Benefits  of  Distributed  Testing,  but  there  were  inadequate  data  to  validate  the  results 
over  a  broad  range  of  applications. 

Concerns  and  constraints  were  primarily  addressed  by  identifying  problems  and  lessons  learned 
in  the  conduct  of  the  three  JADS  tests.  Concerns  and  constraints  are  documented  in  the  test 
phase  reports  and  in  the  networking  and  engineering,  cost  and  benefits  of  distributed  testing, 
verification,  validation,  and  accreditation  (VV&A)  of  distributed  tests,  and  HLA  special  reports. 
Two  categories  of  concerns  and  constraints  were  addressed:  technical  and  programmatic.  In 
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terms  of  sheer  numbers  JADS  was  very  successful  in  identifying  technical  and  programmatic 
concerns  and  constraints.  Because  the  technology  underlying  ADS  is  advancing  so  rapidly,  many 
of  the  concerns  and  constraints  became  outdated.  The  programmatic  concerns  and  constraints, 
however,  are  more  persistent.  In  most  cases  the  programmatic  issues,  such  a  scheduling  and 
security,  are  the  same  as  in  conventional  testing  only  made  more  complex  by  the  addition  of 
ADS.  For  the  three  JADS  tests  the  programmatic  concerns  and  constraints  were  at  least  equal  to 
the  technical  ones  and  may  prove  more  difficult  to  resolve.  The  JADS  charter  did  not  require  us 
to  resolve  all  the  concerns  and  constraints,  only  to  identify  them.  We  were  certainly  able  to  do 
this,  but  we  were  also  able  to  develop  tools  and  techniques  to  resolve  many  of  the  concerns  and 
constraints  that  we  encountered. 

A  general  methodology  that  covered  all  phases  of  a  test  program  from  planning,  development, 
execution,  and  evaluation  was  developed.  The  two  areas  where  the  ADS  methodology  was 
significantly  different  from  a  conventional  test  methodology  were  development  and  execution. 
The  ADS  development  methodology  required  the  addition  of  a  network  design  and  setup 
methodology  and  a  modified  VV&A  methodology.  The  JADS  approach  to  developing  the 
VV&A  methodology  was  to  review  existing  Department  of  Defense  (DoD)  methodologies, 
modify  as  necessary,  apply  to  our  ETE  and  EW  tests,  and  then  update  based  on  our  findings. 
Test  control  during  test  execution  was  another  area  where  a  modified  methodology  was  required, 
except  in  this  case,  no  well-established  DoD  standards  existed  to  start  with.  Different  methods  of 
test  control  were  used  in  the  three  JADS  tests  and  the  results  incorporated  into  the  test  execution 
methodology.  Planning  turned  out  to  be  the  most  difficult  of  the  methodologies.  The  difficulty 
was  in  developing  a  methodology  that  was  general  enough  to  apply  to  all  types  of  DT&E  and 
OT&E  tests  for  all  possible  types  of  military  systems  and  specific  enough  to  provide  the  detail 
needed  for  a  specific  program  to  determine  the  optimum  mix  of  test  methods,  including  ADS,  to 
support  that  program.  Although  the  JADS-developed  methodology  was  a  significant 
accomplishment,  it  should  be  considered  a  work  in  progress  and  not  100  percent  complete. 

The  last  task  in  the  charter  was  to  identify  the  requirements  that  must  be  introduced  into  ADS 
systems  if  they  are  to  support  a  more  complete  T&E  capability  in  the  future.  The  requirements 
were  primarily  derived  from  the  concerns  and  constraints  task.  The  JADS  Final  Report,  utility 
reports,  and  briefings  documented  these  requirements  and  identified  the  appropriate  programs 
and  organizations  for  action. 

JADS  did  not  execute  within  its  original  schedule  or  budget.  Of  the  three  chartered  tests,  the  SIT 
required  restmcturing  because  of  a  12-month  delay  in  getting  a  key  piece  of  software  developed 
by  a  missile  program  office.  The  ETE  Test  execution  had  to  be  delayed  12  months  as  well 
because  of  inadequate  funding  in  the  first  year  of  the  test  and  difficulty  getting  a  contract  vehicle 
to  Northrop  Grumman  set  up.  These  delays  increased  the  requirement  for  infrastructure  and 
contractor  support  in  the  final  year  of  JADS.  Additionally,  a  mid-test  assessment  of  technical 
risk  determined  that  JADS  had  unacceptable  funding  margins  based  on  the  risk  involved  in  the 
remaining  test  activities.  As  a  result,  the  joint  test  director  (JTD)  provided  a  decision  briefing 
requesting  additional  funding  to  the  Deputy  Director  of  Systems  Assessment.  Another  $3.1 
million  was  allocated  in  May  1997. 
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In  spite  of  the  schedule  and  budget  changes,  it  is  the  JTD’s  assessment  that  JADS  was  nearly  95 
percent  successful  in  accomplishing  its  chartered  mission.  The  results  of  the  tests  were  the 
primary  basis  for  addressing  the  three  tasks  in  the  charter.  Present  utility  was  determined  to  be 
case  specific  and  proven  to  exist  for  certain  classes  of  systems  and  suspect  for  others;  concerns, 
constraints,  and  methodologies  as  well  as  requirements  to  facilitate  increased  use  of  ADS  were 
identified  and  published  in  test  and  special  reports.  Inroads  were  made  into  many  test 
organizations  and  acquisition  programs  that  increased  the  use  of  ADS  in  all  facets  of  test  and 
evaluation.  Significant  positive  impact  was  made  by  JADS  on  the  development  of  the  DoD  s 
high  level  architecture  ensuring  the  needs  of  the  T&E  community  were  addressed.  Although 
there  were  many  challenges,  the  work  was  technically  stimulating  and  the  JTF  was  highly 
motivated  to  succeed. 

3.2  Accomplishment  of  JADS  Legacy 

The  JTD  defined  the  JADS  legacy  to  encompass  all  actions  the  JT&E  program  took  to  ensure 
that  its  products  were  fully  incorporated  into  the  user  community.  The  aspects  to  this  approach 
fell  into  three  nominally  sequential  (though  overlapping)  phases.  These  were  (1)  educating  the 
user  community  and  assimilating  ADS  into  its  thought  processes,  (2)  equipping  the  user 
community  with  the  proper  ADS  tools,  procedures,  and  knowledge,  and  (3)  institutionalizing 
JADS’  products,  which  addressed  a  wide  range  of  “things”  from  recommended  directives  to  a 
home  for  the  JADS  library. 

The  education  aspect  was  targeted  at  the  DT&E  and  OT&E  (as  well  as  acquisition  and  industry) 
user  communities  and  was  further  subdivided  into  a  “familiarize”  objective  and  a  consider 
objective.  To  familiarize  the  user  community  with  what  ADS  is,  its  potential  uses  in  the 
T&E/acquisition  processes,  and  the  lessons  learned  from  both  tests  and  other  key  ADS  activities, 
many  tools  were  used.  These  tools  included  the  JADS  newsletter.  Reality  Check',  a  JADS  web 
site;  a  JADS  booth  and  technical  articles  at  major  T&E  symposiums  and  workshops;  JADS 
videos  and  multimedia  compact  disks;  a  JADS  training  course  that  was  given  on  and  off  site; 
JADS  reports;  and  numerous  overview  briefings  given  during  site  visits.  The  JADS  training 
course  was  given  on  site  20  times.  The  course  was  given  off  site  at  locations  such  as  GTRI; 
NAWC-WPNS;  Air  Force  Development  Test  Center  (AFDTC),  Eglin  Air  Force  Base,  Florida; 
U.S.  Army  Test  and  Experimentation  Command  (TEXCOM);  Commander,  Operational  Test  and 
Evaluation  Force  (COMOPTEVFOR);  U.S.  Army  Test  and  Evaluation  Command  (ATEC); 
Operational  Test  Command  (OTC);  U.S.  Army  Developmental  Test  Command  (DTC);  U.S. 
Army  Evaluation  Command  (AEC);  University  of  Texas,  Austin,  Texas;  Boeing  Company, 
Seattle,  Washington;  Fort  Bliss,  Texas;  Pt  Mugu,  California;  MITRE;  and  multiple  International 
Test  and  Evaluation  Association  (TTEA)  conferences  at  Orlando,  Florida;  Las  Cruces,  New 
Mexico;  Fairfax,  Virginia;  Kauai,  Hawaii;  and  Adelaide,  Australia.  Over  1400  people  attended 
these  training  courses.  Although  reaching  all  of  the  user  communities  was  a  daunting  task, 
through  the  use  of  these  multiple  tools  JADS  was  able  familiarize  the  majority  on  the  use  of  ADS 
to  support  T&E. 

The  “consider”  objective  was  to  provide  the  user  community  with  information  or  “evidence  that 
ADS  has  sufficient  utility  for  T&E  to  consider  using  it  on  future  programs.  JADS  provided  a 
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significant  amount  of  evidence  in  our  phase,  class  and  final  reports  and  associated  briefings.  In 
addition,  JADS  targeted  several  programs  in  each  of  the  services  that  were  in  the  early  planning 
stage  and  assisted  them  in  considering  the  use  of  ADS.  Because  of  the  long  lead  times  required 
to  plan  and  implement  a  test  program,  it  will  be  several  years  before  the  final  assessment  of  this 
objective  can  be  made. 

The  objective  of  the  equipping  aspect  of  legacy  was  to  provide  the  user  and  implementer 
communities  with  the  methodologies,  procedures,  knowledge,  and  tools  needed  to  successfully 
implement  ADS  in  their  particular  domain/application.  For  the  user  community,  the  JADS 
approach  was  to  develop  class  reports  that  were  domain/application  specific  and  to  develop  a  test 
planning  methodology  that  included  the  use  of  ADS,  and  to  provide  ADS  training  courses.  The 
approach  for  implementers  was  to  participate  in  service  and  OSD  working  groups,  to  develop 
development,  execution,  and  evaluation  methodologies,  to  develop  software  tools,  and  to  provide 
ADS  training  courses.  Although  the  technology  and  the  acquisition  process  it.self  are  rapidly 
changing,  JADS  was  successful  in  equipping  the  T&E  users  and  implementers  with  the  basics 
necessary  for  them  to  proceed. 

The  JTD  saw  three  major  objectives  in  the  institutionalize  aspect  of  legacy:  (1)  recommend 
policies  and  directives  to  facilitate  the  incorporation  of  ADS  into  the  T&E  process;  (2)  find  and 
successfully  access  the  appropriate  repositories  for  JADS  data  and  knowledge;  and  (3)  find  and 
successfully  access  the  appropriate  final  homes  for  JADS-developed  products  and  equipment. 

The  approach  to  the  first  of  these  objectives  was  to  review  current  OSD  and  service  initiatives, 
roadmaps,  master  plans,  and  policy  guidance  relevant  to  the  use  of  ADS  to  support  T&E  and  to 
then  provide  feedback  to  the  respective  originators  on  recommended  changes  that  would 
facilitate  its  use.  This  was  successfully  accomplished. 

Another  aspect  of  institutionalization  efforts  was  participation  in  the  key  professional  society  for 
distributed  simulation  and  DoD’s  HLA  efforts.  JADS  took  several  major  leadership  roles  in  the 
transition  of  the  training-oriented  DIS  Workshops  to  the  training-,  acquisition-  and  analysis- 
oriented  Simulation  Interoperability  Workshops  (SIW)  and  the  creation  of  the  Simulation 
Interoperability  Standards  Organization  (SISO).  In  the  HLA  area,  JADS  became  a  sitting 
member  of  the  Architecture  Management  Group  (AMG)  and  provided  the  only  test  and 
evaluation  experimentation  with  early  versions  of  AMG  products. 

A  survey  of  OSD  and  service  repositories  was  conducted,  and  homes  were  found  for  all  JADS 
reports,  lessons  learned,  and  test  data.  In  most  cases,  multiple  homes  were  found  for  each  item, 
so  this  objective  was  successfully  addressed. 

The  final  objective  was  to  find  final  homes  for  JADS-developed  products  and  equipment.  The 
JADS  transition  plan  documents  the  successful  accomplishment  of  this  objective. 

Overall,  the  JADS  legacy  program  was  an  unqualified  success.  Newsletters,  conference 
participation,  web  site,  “ADS  for  T&E”  professional  development  course,  lessons  learned  videos 
and  multimedia  report  formats  allowed  JADS  to  reach  and  influence  a  broad  swath  of  the  T&E 
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community,  both  government  and  civilian.  The  T&E  community  was  educated  to  at  least 
consider  using  ADS  to  support  future  test  programs.  Testers  were  equipped  with  the  basics 
needed  to  successfully  plan  and  execute  an  ADS  test,  and  JADS’  products  were  institutionalized 
to  provide  a  firm  foundation  on  which  to  build.  Success  did  not  come  easily  in  legacy.  The 
tendency  of  the  JTF  was  to  focus  on  the  successful  execution  of  the  three  tests.  Developing  a 
legacy  plan  early  in  the  program  and  assigning  a  high  priority  to  legacy  (including  a  full-time 
legacy  manager)  as  well  as  committing  much  of  the  service  deputies’  time  to  legacy  activities 
were  critical  to  this  effort.  Subsequent  “nagging” -was  required  by  the  JADS  staff  to  ensure 
legacy  actions  were  occurring  early  enough  and  long  enough  to  have  an  impact  on  the  T&E 
community.  Finally,  many  trips  were  required  to  get  the  users’  attention  through  face-to-face 
meetings. 
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4.0  Lessons  Learned  Overview 


M^ny  lessons  were  learned  during  the  conduct  of  the  JADS  program.  The  following  represent 
those  deemed  most  helpful  for  the  management  of  future  JT&E  programs.  The  lessons  learned 
were  developed  by  the  team  leads  or  personnel  in  charge  of  the  area  and  have  been  provided  with 
only  minimal  editing.  The  lessons  have  been  arranged  into  fourteen  categories. 

4.1  Organizational  Structure 

•  A  matrix  organization  provided  the  JTF  the  ability  to  flex  resources  easily  to  meet  the 
needs  of  each  individual  test  and  the  JTF  as  a  whole. 

JADS  organized  itself  as  a  matrix  organization  primarily  because  it  was  going  to  run  three 
distinctly  different  test  programs  that  would  overlap  in  terms  of  planning,  execution  and 
reporting.  Two  teams  provided  the  bulk  of  direct  support  to  the  three  test  teams.  The  Network 
and  Engineering  (N&E)  team  provided  all  networking  support  for  the  tests  and  developed  and 
managed  our  Test  Control  and  Analysis  Center  (TCAC).  The  Analysis  team  provided  analysis 
support  to  each  test  team  and  was  responsible  for  overall  JADS  issues  and  objectives. 

The  concept  of  matrixing  the  N&E  and  analyst  resources  was  decried  by  the  test  team  leads  as 
keeping  them  from  getting  the  support  they  needed.  However,  as  the  test  program  went  on,  test 
teams  became  more  adept  at  coordinating  their  requirements  for  support,  and  the  knowledge 
gained  by  the  matrix-support  teams  from  previous  testing  greatly  enhanced  the  execution  of  each 
test  event.  Using  this  approach,  it  was  relatively  easy  to  mass  resources  on  the  hot  problem  of 
the  day  or  month. 

The  Support  team  provided  the  typical  administrative  functions  for  all  JADS  personnel.  The 
most  critical  of  these  functions  for  test  success  were  travel  coordination  and  security.  The 
security  noncommissioned  officer  (NCO)  ensured  each  test  team  had  appropriate  security 
classification  guidelines  available  and  reviewed  all  documents  for  both  classification  and 
distribution  compliance.  Having  all  support  functions  provided  by  personnel  assigned  to  the  JTF 
proved  to  be  invaluable.  The  result  was  JADS  personnel  received  the  support  when  needed, 
versus  not  receiving  needed  support  from  an  external  agent. 

The  Program  Control  team  consisted  of  the  JADS  budget  person  (an  Air  Force  (AF)  NCO),  the 
legacy  manager  (GS-13)  and  an  assistant.  Halfway  through  the  test,  JADS  converted  the 
director’s  secretary  position  to  a  technical  editor  position  and  that  position  also  fell  under 
Program  Control.  The  technical  editor  proved  to  be  exceptionally  valuable  to  JADS’  success.  If 
we  were  to  do  JADS  over  again,  we  would  factor  in  a  technical  editor  from  the  start. 

•  Having  a  chief  of  staff  supported  by  Program  Control  and  Support  team  leads  (and 
sometimes  the  executive  officer)  provided  continuity  of  leadership  during  the  many 
temporary  duties  (TDY)  of  the  JTD. 
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About  twelve  months  after  chartering,  it  became  apparent  to  the  joint  test  director  that,  because 
of  the  demanding  travel  schedule,  JADS  needed  someone  “in  charge”  on  a  daily  basis  when  the 
JTD  was  not  available.  Consequently,  the  director  established  the  position  of  chief  of  staff  that 
became  an  additional  assigned  duty  for  one  of  the  service  deputies.  The  chief  of  staff  was 
explicitly  not  put  in  the  direct  line  of  supervision  between  the  team  leads  and  the  director  and 
essentially  became  a  facilitator  for  all  JT&E  operations.  This  approach  maintained  the 
responsibility  and  authority  of  the  team  leads  that  might  have  been  lost  if  the  position  of  deputy 
test  director  had  been  established  instead.  The  only  person  rated  by  the  chief  of  staff  was  the 
executive  officer,  a  position  only  formally  established  periodically  based  on  available  personnel. 
For  about  18  months,  the  senior  enlisted  advisor  performed  executive  officer  functions.  Then  a 
major  was  assigned  as  both  Program  Control  and  Support  team  lead  and  also  performed 
executive  officer  duties.  Finally  a  captain  was  assigned  executive  officer  and  legacy  duties.  The 
person  serving  as  executive  officer  assisted  the  chief  of  staff. 

•  A  formal  review  body  manned  by  JTF  leadership  (JADS  formed  a  steering  committee) 
greatly  facilitated  resolution  of  technical  and  programmatic  issues  and  improved 
communication  flow  among  the  key  members  of  the  JTF. 

Within  six  months  after  the  arrival  of  all  the  service  deputies,  it  became  apparent  that  there  was 
no  mechanism  to  enable  the  cross-flow  of  information  among  teams  and  between  the  teams  and 
JT&E  management  (other  than  the  director).  This  lack  of  cross-flow  led  to  resource  conflicts 
within  the  matrix  organization  and  to  the  loss  of  focus  on  the  “big  picture.”  To  resolve  this 
problem,  JADS  established  a  steering  committee  to  serve  as  the  vehicle  for  information 
exchange.  Membership  consisted  of  the  service  deputies,  the  technical  advisor  and  the  principal 
investigator  for  the  support  contractor  (the  “Big  Five”)  and  all  team  leads.  The  director  was 
explicitly  left  off  the  membership,  though  included  in  the  steering  committee  E-mail  group.  This 
was  to  allow  for  free  and  open  discussion  by  the  group,  which  could  be  handicapped  by  the 
Director's  presence.  The  Director  occasionally  called  meetings  of  the  steering  committee,  which 
he  would  preside  over,  to  address  specific  issues. 

Steering  committee  meetings  were  chaired  by  the  chief  of  staff  and  could  be  called  by  any 
member.  Meetings  focused  on  programmatic  and  technical  issues  associated  with  JADS  tests 
and  were  limited  to  two  hours.  (If  there  was  unfinished  business,  another  meeting  was 
scheduled.)  “Big  Five”  members  were  required  to  attend  all  meetings.  Other  steering  committee 
members  could  attend  if  they  desired.  A  charter  was  established  for  the  steering  committee  that 
allocated  no  decision-making  authority  to  the  committee  itself;  however,  all  the  decision  makers 
in  JADS  except  for  the  director  were  members  of  the  committee.  Consequently,  the  steering 
committee  became  a  ‘coercive’  body  designed  to  establish  consensus  or  identify  disagreements 
that  could  not  be  resolved.  Another  major  function  of  the  steering  committee  was  the  editorial 
review  of  all  JADS  documents,  briefings  and  technical  papers.  As  many  issues  as  possible  were 
resolved  via  E-mail  discussions.  The  products  of  these  steering  committee  meetings  were 
normally  options  and  recommendations  provided  to  the  Director  for  his  action. 

•  Having  the  entire  test  team  collocated  contributed  to  its  efficiency  and  success. 
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Despite  the  distributed  nature  of  JADS’  program,  all  personnel  were  housed  under  one  roof. 
Additionally,  contractor  personnel  were  integrated  into  functional  or  matrix  teams,  based  on  their 
expertise.  This  eliminated  any  “us  versus  them”  mentality.  The  result  was  a  high  degree  of 
cohesion  as  an  organization,  synergy  off  the  skills  of  one  another,  better  communication,  and 
more  efficient  mission  accomplishment. 

4.2  Analysis 

•  A  matrix  organization  structure  provided  synergistic  effects,  objectivity  and 
independence  in  the  area  of  analysis. 

The  Analysis  team  provided  support  to  each  of  the  individual  test  teams  and  was  also  responsible 
for  answering  overall  JADS  issues  and  objectives.  The  Analysis  team  conducted  the  crosswalk 
of  test  MOEs/MOPs  with  the  JADS  MOEs/MOPs  and  monitored  other  programs  using  ADS  so 
JADS  could  extend  its  findings  beyond  the  scope  of  our  three  individual  tests.  Each  test  team 
had  an  attached  analyst,  whose  primary  job  was  to  coordinate  support  requirements  with  the 
corresponding  matrix-support  team.  This  organization  facilitated  the  daily  sharing  of  ideas  and 
lessons  learned,  which  ensured  that  all  the  analysts  stayed  informed  and  kept  abreast  of  each  of 
the  three  tests.  The  cross-fertilization  of  ideas,  intellectual  discussions  and  collective  approach  to 
solving  problems  helped  broaden  the  analysts’  perspectives.  It  better  prepared  the  analysts  to 
support  their  individual  test  team  as  well  as  to  address  broader  overall  JADS  issues  and 
objectives.  A  matrix  Analysis  team,  independent  of  the  test  teams,  also  helped  ensure  thorough 
and  completely  objective  analysis.  Individual  test  team  leads  were  responsible  for  the  planning 
and  management  of  limited  resources  to  execute  their  tests.  Since  support  analysts  were  not 
directly  assigned  to  the  individual  test  teams,  they  could  focus  completely  on  the  thorough  and 
objective  analysis  and  reporting  of  test  results.  Had  they  been  assigned  directly  to  the  individual 
test  teams,  there  may  have  been  a  tendency  to  refocus  analysis  efforts  onto  more  immediate  test 
execution  concerns  at  the  expense  of  less  immediate  data  management  and  analysis  requirements. 
The  Analysis  team’s  independence  and  objectivity  led  to  more  reliable  and  credible  results. 

•  A  distributed  test  architecture  required  unique  network  analysis  capabilities. 

The  distributed  nature  of  the  tests  and  the  JADS  charter  to  address  the  utility  of  ADS  for  T&E 
necessitated  unique  network  analysis  capabilities.  We  dedicated  one  analyst  to  support  each  of 
the  test  teams  including  the  N&E  team  in  this  effort.  Although  a  member  of  the  N&E  team  may 
have  brought  more  detailed  computer  knowledge  and  experience  to  the  network  performance 
evaluation  process,  great  benefit  was  achieved  by  having  an  Analysis  team  member  perform  this 
role.  The  network  analyst  worked  closely  with  members  of  each  test  team  to  become  familiar 
with  each  specific  network  architecture  and  with  the  team's  issues  and  concerns  for  the  network 
in  being  able  to  satisfactorily  support  the  collection  of  quality  system  under  test  data.  The 
network  analyst  then,  with  N&E  assistance,  directed  network  monitoring  and  characterization 
activities  to  determine  the  impact  of  network  performance  issues  on  the  quantity  and  quality  of 
data  collected.  The  network  analyst  also  ensured  satisfactory  dissemination  of  conclusions  back 
to  test  team  analysts.  JADS  chose  the  Cabletron  System’s  SPECTRUM®  network  analysis 
package  as  its  primary  tool  for  real-time  network  traffic  monitoring.  Proficient  use  of 
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SPECTRUM®  required  approximately  3  days  of  vendor  training  and  an  underlying  knowledge  of 
basic  UNIX^*^  commands.  Network  analysis  requires  some  in-depth  understanding  of  data 
communications  processes,  network  equipment  and  protocols,  local  area  network  (LAN)  and 
wide  area  network  (WAN)  technologies,  and  network  performance-monitoring  techniques.  This 
knowledge  can  be  obtained  via  any  training  class  that  offers  an  overview  of  computer  networking 
fundamentals. 

•  The  Analysis  team  became  the  de  facto  operational  reserve  for  providing  replacement 
personnel  to  the  test  teams,  which  negatively  impacted  analysis  of  JADS  issues. 

As  alluded  to  above,  each  of  the  test  team  leads  was  charged  with  the  responsibility  to  plan, 
execute  and  manage  the  test  with  limited  personnel  resources.  As  these  teams  experienced 
personnel  turnover,  test  execution  schedules  dictated  that  personnel  be  replaced  immediately.  To 
meet  the  immediate  need  for  replacements,  experienced  analysts  with  a  broad  understanding  of 
JADS  analysis  requirements  were  reassigned  to  an  individual  test  team  to  fulfill  other  than 
analysis  roles  on  the  team.  When  new  personnel  eventually  arrived  they  backfilled  the  analysis 
team.  These  personnel  had  to  be  trained  and  brought  up  to  speed  on  JADS.  They  lacked  the 
experience  and  depth  of  understanding  concerning  the  analysis  of  the  JADS  issues  and 
objectives. 

4.3  Network  and  Engineering 

The  mission  of  the  Network  and  Engineering  team  was  to  provide  communications  support  to  the 
JADS  JTF.  This  included  providing  a  LAN  for  50  users  and  a  multisecurity  network  for  the 
TCAC  that  supported  three  tests.  N&E  was  also  responsible  for  determining  the  requirements 
for  each  test  team,  purchasing,  installing  and  maintaining  the  equipment,  and  requesting, 
installing  and  maintaining  all  of  the  long  haul  circuits. 

•  Determine  whether  your  requirements  are  to  be  supported  by  your  organization  or 
another  and  the  kind  of  support  to  be  provided. 

As  mentioned  previously,  JADS  benefited  significantly  by  having  its  support  internal  to  the  JTF. 
Types  of  support  to  be  considered; 

Voice  telephone  systems  LAN  support 

LAN  Test  control  facility 

WAN  Classified  facility 

Communications  security  (COMSEC)  account  Multisecured  facility 

Procurement  of  equipment  Request  for  circuits 

Software-specific  needs  Time  synchronization 

Inter-Range  Instrumentation  Group  (IRIG)/Global  Positioning  System  (GPS) 

•  A  blend  of  contractor  and  military  personnel  was  required  to  support  all  the  above 
requirements. 
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In  your  request  for  personnel,  you  have  to  be  very  specific  as  to  the  kind  of  person  needed  (Air 
Force  specialty  code  [AFSC]  or  the  Army  military  occupational  specialty  [MOS].)  On  the 
contractor  side,  a  minimum  of  network  engineering  (local  and  wide  area  networks)  and  hardware 
and  software  engineering  capability  would  be  required.  Getting  the  right  people  and  having  them 
available  at  the  start  of  the  program  is  the  most  crucial  step  for  N&E. 

•  Early  requirements  deflnition  and  site  surveys  for  networking  support  are  critical. 

Requirements  have  to  be  identified  by  the  test  teams  and  presented  to  the  N&E  team.  Both  teams 
then  need  to  discuss  the  options  for  supporting  those  requirements  and  to  develop  a  scheduled 
plan  for  purchasing,  test  bedding,  and  installing  the  equipment.  Site  surveys  are  very  important 
and  need  to  be  conducted  by  representatives  from  both  teams.  All  this  has  to  be  done  in  the  early 
stages  of  your  planning. 

•  Personnel  need  to  understand  the  time  constraints  for  getting  required  support. 

For  sole-ownership  of  long-haul  circuits,  organizations  must  submit  an  exception  to  policy  to 
Defense  Information  System  Agency  (DISA)  180-210  days  prior  to  the  test  date. 

Circuit  requests  require  a  minimum  of  150-180  days  depending  on  the  base  procedures  for 
requesting  a  circuit. 

Establishing  a  COMSEC  account  requires  a  minimum  of  180  days  for  training  personnel  and 
certification. 

Ordering  COMSEC  material  requires  a  minimum  of  120  days. 

Ordering  equipment  varies  with  the  method  of  purchase: 

-  Credit  card  limit  of  $  2,500  allows  local  purchase  or  mail  order  items  within  a  matter  of 
days. 

-  Contract  avenues  vary  anywhere  from  two  weeks  to  several  months  depending  on  the 
specific  contract  and  the  type  of  equipment  being  purchased. 

-  Going  through  base  procurement  can  take  anywhere  from  three  to  eight  months. 

Units  should  establish  an  accredited  classified  facility  as  soon  as  possible  to  allow  for  corrective 
actions  and  to  have  the  capability  to  conduct  risk  reductions  prior  to  testing. 

•  JT&E  use  of  common  user  telecommunications  services  is  now  mandatory  unless 
waivers  are  obtained. 
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Assistant  Secretary  of  Defense  (ASD)  (C3I)  Policy,  5  May  97  mandated  use  of  Defense 
Information  Systems  Network  (DISN)  or  Federal  Telecommunications  System  (FTS)  common 
user  telecommunications  services  for  all  DoD  long-haul  telecommunication  requirements. 
Waivers  or  exceptions  to  policy  are  granted  only  under  extraordinary  circumstances.  Detailed 
information  of  your  requirements  has  to  be  presented  to  DISA  for  them  to  evaluate  whether  their 
network  could  or  could  not  support  your  requirements. 

Send  exception  requests  to 

E-mail:  D3WAIVE@ncr.disa.mil 

DISA  HQD312 
Courthouse  Road 
Arlington  VA  22204-2199 

4.4  Support 

•  Structure  Support  team  based  on  the  structure  of  the  JTF. 

The  Support  team  was  comprised  of  one  Army  personnel  specialist,  one  Air  Force  personnel 
specialist,  one  Air  Force  supply  specialist,  and  one  Air  Force  administrative  specialist.  Support 
provided  the  typical  administrative  functions  for  all  JADS  personnel.  As  mentioned  before,  the 
most  critical  of  these  functions  for  test  success  were  travel  coordination  and  security.  The 
administration  specialist  was  also  the  security  NCO  and  ensured  all  security  issues  were  properly 
addressed.  The  Support  team  also  conducted  typical  orderly  room  functions  such  as  weight 
control  management,  physical  fitness  testing,  awards  and  decorations,  retirements,  etc. 

There  were  13  Army  authorizations  and  19  Air  Force  authorizations,  so  it  was  determined  early 
during  the  feasibility  study  that  both  an  Army  and  Air  Force  personnel  specialist  would  be 
needed.  The  Air  Force  personnel  specialist  arranged  travel  coordination  for  all  JADS  personnel 
minus  contractors  and  also  acted  as  the  director’s  secretary.  It  was  also  determined  that  JADS 
needed  a  supply  specialist  because  with  distributed  tests  our  equipment  would  be  located 
throughout  the  United  States.  Finally,  an  administrative  specialist  was  needed  to  handle 
administrative  tasks  and  also  act  as  security  NCO.  Another  important  service  that  the 
administrative  NCO  provided  was  to  lead  the  archiving  effort  at  the  end  of  the  test. 

•  Have  a  good  plan  for  archiving  information  and  data  and  begin  early. 

There  are  two  joint  test  and  evaluation  archiving  repositories:  the  SAIC  Technical  Library  in 
Alexandria,  Virginia,  and  Headquarters  Air  Force  Operational  Test  and  Evaluation  Center, 
History  Office  (HQ  AFOTEC/HO)  Technical  Library  located  on  Kirtland  Air  Force  Base,  New 
Mexico.  These  two  repositories  maintain  historical  joint  test  and  evaluation 
documentation/records  and  provide  copies  to  authorized  requesters.  The  two  repositories 
maintain  records  only  up  to  the  secret  level. 
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The  types  of  records  archived  are  charters,  directives,  plans,  reports,  pertinent  high-level 
correspondence,  key  briefings,  and  relevant  visual  information  (videos,  tapes,  slides,  films  and 
photographs).  Raw  data  in  any  form,  routine  administrative  material,  or  any  information 
classified  higher  than  secret  are  not  archived.  A  detailed  archiving  list  can  be  obtained  from  the 
HQ  AFOTEC  Technical  Library  repository. 

Standard  Form  (SF)  135,  Records  Transmittal  and  Receipt,  was  used  for  archiving  records. 
Required  information  includes  the  year  of  items  being  archived;  volume  of  records  in  cubic  feet 
(per  box);  a  complete  series  description  with  inclusive  dates;  disposal  authority  (JADS  JTF  used 
Air  Force  Manual  [AFMAN]  37-139,  Records  Disposition  Schedule,  AFOTEC  Instruction  61- 
204,  Disseminating  Scientific  and  Technical  Information,  AFOTEC  Instruction  84-101, 
Requirements  for  OT&E  Program  Case  Files  and  Other  Historical  Information,  and  HQ 
AFOTEC  letter  guidance  on  OT&E  and  research  and  development  program  case  files);  and  a 
proper  disposal  date  for  the  records. 

For  shipping  and  storage  of  the  records,  specific  archiving  boxes  were  used.  These  boxes  were 
numbered  consecutively  with  the  total  number  being  shipped  (example,  1  of  15).  Classified 
records  are  not  shipped  together  in  the  same  boxes  with  unclassified  records.  Use  separate  boxes 
for  classified  records  (secret  and  confidential  may  be  shipped  and  stored  together.)  Mark 
classified  records  with  the  highest  classification  they  contain,  and  mark  the  box  with  the  highest 
classification  it  holds.  Do  not  attach  SF  704,  Secret  Cover  Sheet,  or  SF  705,  Confidential  Cover 
Sheet,  to  any  classified  records  being  archived.  Follow  DoD  Regulation  5200.1,  Information 
Security  Program,  when  preparing  classified  material  for  transmission. 

NOTE:  Copies  of  some  documentation  such  as  plans,  test  reports,  and  other  documents  of 
significant  technical  value  completed  by  the  joint  test  force  were  sent  to  the  Defense  Technical 
Information  Center  (DTIC)  and  the  Modeling  and  Simulation  Information  Analysis  Center 
(MSIAC).  Both  provide  authorized  copies  (for  a  fee)  of  JT&E  reports  and  other  technical 
documents  that  they  have  stored  in  their  optical  databases  upon  request.  DTIC  and  MSIAC  hold 
records  only  up  to  the  secret  level. 

Addresses: 

SAIC  Technical  Library 

2001  North  Beauregard  St.  Suite  800 

Alexandria  VA  22311 

HQ  AFOTEC/HO 
Technical  Library 
8500  Gibson  Blvd  SE 
Kirtland  AFB  NM  87117-5558 

Defense  Technical  Information  Center  (DTIC) 

8725  John  J.  Kingman  Road,  Suite  0944 
Ft  Bel  voir  VA  22060-6218 
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Modeling  and  Simulation  Information  Analysis  Center  (MSIAC) 

1901  N  Beauregard  St.,  Suite  400 
Alexandria,  VA  22311-1705 

4.5  Contractor  Support 

•  Because  of  the  fluid  nature  of  JT&E  issues  and  the  uncertainty  of  many  leveraged’  test 
events,  on-site  contractor  support  was  preferable  to  any  other  alternative. 

Science  Applications  International  Corporation  (SAJC)  provided  on-site  contract  support.  The 
S  AIC  contract  support  team,  on  average,  included  a  program  manager  (also  performed  as  a  senior 
engineer),  a  senior  scientist,  two  senior  engineers,  three  senior  software  engineers,  three  senior 
analysts,  a  graphic  illustrator,  two  analysts,  a  network  technician,  and  an  administrative  assistant. 
The  program  manager  was  responsible  for  staffing  the  SAIC  technical  support  staff  and  arranging 
for  outside  assistance  when  required.  Until  the  final  phase-down  began,  the  on-site  support 
remained  steady  at  about  fifteen  man-years  per  year.  Occasionally  specialized  SAIC  assistance 
was  brought  in  to  support  specific  program  technical  needs.  The  SAIC  support  was  fully 
integrated  into  each  of  the  three  test  teams,  the  network  team,  and  the  analysis  team.  Contract 
relations  and  deliverables  were  designed  to  allow  free  crossflow  of  information  between  all 
parties.  The  result  was  a  highly  effective  working  environment  where  there  was  only  minimal 
differentiation  between  government  and  contractor  personnel. 

The  Georgia  Tech  Research  Institute  (GTRI)  provided  three  to  four  man-years  per  year  off-site 
technical  support  for  some  aspects  of  the  Electronic  Warfare  Test. 

The  preponderance  of  senior  engineers  and  analysts  proved  necessary  over  the  course  of  the 
program,  as  did  the  on-site  approach  to  contracted  support.  The  nature  of  the  test  planning, 
design,  and  implementation  tasks  required  people  with  high  levels  of  experience  in  T&E.  The 
on-site  approach  proved  crucial  given  the  innovative  nature  of  the  work,  the  requirement  for 
minute-by-minute  technical  coordination  between  government  and  contractor  personnel,  and  the 
many  changes  in  planning  direction  as  technical  surprises  occurred. 

The  contractor/government  mix  worked  well  for  this  program,  and  the  level  of  support  was 
sufficient  to  accomplish  the  program  objectives. 

4.6  Manpow^er/Personnel 
4.6.1  General  Personnel  Issues 

•  Expect  to  take  9-12  months  to  fill  personnel  requirements. 

Once  JADS  was  chartered  in  October  1994,  a  major  effort  ensued  to  ensure  all  requisitions  with 
the  appropriate  service  agency  were  on  file  to  begin  identifying  personnel  who  would  initially  fill 
the  vacant  manpower  authorizations. 


Two  civilians,  who  were  assigned  to  the  feasibility  study,  were  ultimately  permanently  assigned 
to  the  newly  chartered  test  force.  This  provided  much  needed  continuity.  On  the  other  hand, 
identifying  qualified  military  personnel  as  “best  fit”  for  a  particular  position  was  difficult. 

The  director  was  the  final  authority  in  determining  whether  or  not  an  individual  was  suitable  to 
fill  a  position.  For  the  most  part.  Air  Force  personnel  had  the  needed  credentials,  but  it  was 
difficult  and  often  impossible  to  identify  an  “available”  and  qualified  Army  asset.  This 
necessitated  the  selection  of  a  less-than-qualified  person  and  then  sending  that  person  to  school 
for  training. 

Navy  personnel  were  extremely  difficult  to  obtain,  as  is  usually  the  case  for  JT&Es.  The  direct 
involvement  of  the  Commander  of  the  OPTEVFOR  was  the  reason  JADS  received  its  first  two 
Navy  personnel. 

Arrival  dates  for  inbound  personnel  were  the  next  major  hurdle.  Many  individuals  wanted  to 
wait  until  the  summer  months  to  report.  This  was  in  large  part  due  to  family  issues  that  involved 
children  being  in  school.  As  much  as  possible,  the  director  accommodated  the  service  person’s 
desires. 

In  the  end,  it  took  between  nine  and  twelve  months  from  the  JTF  chartering  to  when  sufficient 
numbers  of  personnel  had  arrived  to  begin  test  execution  and  other  mission-related  activities. 

•  Get  the  right  skill  sets  on  board  early,  to  include  people  with  practical  T&E  experience. 

Because  of  the  slow  nature  of  manning  the  JTF,  it  is  important  to  get  the  right  types  of  people  on 
board  as  soon  as  possible.  This  can  be  done  to  a  limited  extent  via  military  and  government 
manning.  More  likely,  you  will  need  to  rely  on  your  support  contractor  to  front-load  skills  you 
identify  as  those  you  must  have  soonest.  Don’t  neglect  people  with  practical  T&E  experience  in 
this  step,  as  they  will  pay  big  dividends  in  completing  the  Program  Test  Plan  and  readying  you 
for  your  early  test  events. 

•  Adequately  manned  and  centralized  administrative  support  was  a  key  to  JADS  success. 

The  test  teams’  needs  were  satisfied  from  an  operational  perspective,  but  they  lacked 
administrative  support.  There  were  basically  two  choices  in  an  effort  to  satisfy  this  requirement: 
centralized  or  decentralized  administrative  support.  Decentralized  support  would  have  given 
each  test  team  a  dedicated  individual  to  meet  the  administrative  needs:  typing,  filing,  travel 
orders  preparation,  reproduction  of  reports,  supply  functions  and  other  miscellaneous  needs. 
However,  it  was  determined  that  a  centralized  administrative/personnel  function  would  best  suit 
the  needs  of  the  entire  JTF  as  a  whole.  This  proved  to  be  most  successful  because  it  precluded 
“one-deep”  slots  and  provided  flexibility  when  workloads  became  heavy  and  demanding.  The 
centralization  of  all  administrative  functions  also  provided  an  avenue  which  facilitated  cross¬ 
utilization  training  and  precluded  any  administrative  “down  time”  because  all  personnel  knew 
one  another’s  areas  of  responsibility  which  included  Army/Air  Force  administration/personnel 
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processing,  supply,  security,  safety,  report  reproduction,  travel  functions  and  a  knowledge  of  the 
points  of  contact  from  outside  activities  and  supporting  agencies. 

•  Personnel  turnover  will  be  greater  than  you  expect  and  will  require  close  management. 

The  initial  charter  reflected  that  the  existence  of  the  JTF  would  span  a  period  of  five  years. 
However,  with  the  eventual  approval  of  the  senior  advisory  council  that  the  Electronic  Warfare 
Test  be  chartered  as  an  addendum  to  the  original  charter,  the  length  of  the  JTF  was  increased  to 
five  years  and  six  months.  Maintaining  qualified  personnel  in  sufficient  numbers  became  an 
issue  of  concern. 

Because  of  the  difficulty  in  obtaining  personnel  with  the  necessary  qualifications,  JADS  felt 
secure  that  these  individuals  would  be  available  for  a  minimum  of  three  years.  From  an 
operational  perspective  this  was  true,  but  the  personal  side  of  things  was  never  given  much 
thought  or  consideration.  Some  individuals  initially  assigned  had  completed  20  years  of  active 
service  and  became  eligible  for  retirement.  Several  individuals  did  eventually  retire  before  the 
end  of  testing.  To  complicate  matters,  the  Army  and  Air  Force  began  to  offer  early  separation 
and  retirement  options,  and  there  were  several  individuals  who  took  advantage  of  these 
opportunities.  To  make  matters  even  worse,  the  needs  of  the  Army  mandated  a  permanent 
change  of  station  for  several  assigned  individuals  who  departed  as  early  as  after  one  year.  And 
finally,  there  were  others  who,  because  of  career  progression  concerns,  felt  it  was  in  their  best 
interest  to  apply  for  other  assignments. 


Although  some  of  the  transitions  were  identified  far  enough  in  advance  to  allow  the  JTF  to 
adequately  plan  for  the  turnover,  others  were  not,  and  these  resulted  in  manpower  shortages. 
Requisitions  needed  to  once  again  be  initiated,  and  then  once  an  available  person  to  fill  a  position 
was  identified,  it  was  a  period  of  between  four  and  six  months  before  their  arrival.  As  JADS 
moved  into  its  last  two  years  of  existence,  it  became  increasingly  hard  to  acquire  military 
personnel  to  fill  vacancies.  We  were  able  to  offset  some  of  the  problems  through  additional 
contractor  support,  however,  there  is  a  limit  to  the  amount  of  contractor  support  any  JTF  can 
afford.  The  contract  support  team,  if  well  organized,  can  provide  some  degree  of  stabilization 
and  also  be  responsive  to  changing  technical  requirements. 

4.6.2  Army  Personnel 

Coordinate  closely  with  Army  Test  and  Evaluation  Command  (ATEC)  during  development 
of  the  Table  of  Distribution  and  Allowances  (TDA). 

The  TDA  provides  requirements  for  Army  personnel  positions  and  is  managed  by  ATEC.  It  is 
not  part  of  the  ATEC  TDA.  Deputy  chief  of  staff  for  operations  and  plans  (DCSOPS) 
coordinates  the  establishment  of  a  specific  joint  test  TDA  with  the  Army  based  on  input  from  the 
JFS  team.  It  is  a  good  idea  for  feasibility  study  teams  to  get  input  from  serving  JT&E  Army 
deputies  when  putting  together  their  TDA  requests.  Once  a  TDA  request  is  provided  to  ATEC, 
the  Joint  test  manager  at  ATEC  will  submit  the  TDA  for  approval  in  August  of  each  year. 
Timing  is  critical  to  ensure  the  TDA  is  approved  prior  to  JT&E  chartering  so  that  personnel  can 
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be  requisitioned  immediately  after  chartering.  Changes  can  be  made  to  the  joint  test  TDA,  but 
they  can  only  be  approved  in  the  August  cycle. 

•  During  the  majority  of  JADS’  lifetime,  the  Army  manned  joint  tests  at  nearly  100 
percent  and  ATEC  generated  personnel  requisitions. 

Until  just  recently,  the  Army  manned  JT&Es  at  or  near  100%.  Be  advised  that  the  Army  is 
currently  not  doing  so  at  least  for  the  near  term.  The- following  paragraphs  were  written  based  on 
JADS’  experience,  when  the  level  of  Army  manning  was  not  an  issue. 

TDA  requirements  for  enlisted  personnel  equate  to  authorizations  for  enlisted  personnel.  Once 
the  TDA  is  approved,  ATEC  must  submit  a  requisition  to  Army  Personnel  Command 
(PERSCOM)  to  have  the  position  filled.  Minimum  fill  time  for  enlisted  personnel  is 
approximately  six  months  after  the  requisition  is  submitted. 

TDA  requirements  for  officers  do  not  equate  to  authorizations.  The  Army  has  more  officer 
requirements  than  officers  and  fills  requirements  based  on  an  officer  distribution  plan  (ODP)  that 
fills  different  units  at  different  percentages  of  fill.  While  ATEC  is  filled  at  only  60-80  percent, 
the  joint  test  TDA  is  filled  at  approximately  100  percent.  The  joint  test  manager  at  ATEC  is 
responsible  for  coordinating  a  joint  test  ODP  and  allocating  an  ODP  among  the  various  chartered 
joint  tests.  An  ODP  is  provided  by  specialty  and  rank,  and  once  you  have  a  TDA  requirement 
and  an  ODP,  ATEC  can  requisition  personnel  from  PERSCOM.  Minimum  fill  time  for  officers 
is  also  approximately  six  months  after  the  requisition  is  submitted. 

•  Army  deputies  should  directly  coordinate  vrith  PERSCOM  assignment  officers/NCOs 
when  fills  or  backfills  are  required  or  extensions  are  appropriate. 

The  Army  deputy  should  coordinate  directly  with  PERSCOM  assignment  officers  to  ensure 
appropriate  personnel  are  assigned.  Assignment  officers  will  have  little  or  no  clue  about  what  a 
joint  test  is,  what  it  does  or  where  it  is  located.  In  JADS,  we  had  to  get  backfills  for  personnel 
who  retired  or  left  service  and  those  who  needed  to  depart  JADS  for  career  development.  Close 
coordination  with  assignment  officers  as  early  as  possible  makes  for  a  more  rapid  replacement  of 
personnel.  Extensions  for  half  of  our  officers  were  coordinated  through  ATEC  (ATEC 
commanding  general  [CG]  endorsed  all  requests)  and  approved  by  PERSCOM.  Stating  that  an 
ODP  existed  for  the  positions  and  a  backfill  would  be  required  if  the  extension  was  not  approved 
were  key  to  getting  the  extensions  approved. 

•  Stabilize  Army  enlisted  personnel  for  three  years. 

Little  could  be  done  to  prevent  individuals  from  retiring,  but  necessary  measures  were  taken  to 
stabilize  all  other  personnel.  This  was  an  issue  that  emerged  from  the  onset  of  the  organization. 
There  was  no  continuity  of  tour  stabilization  among  the  Army  officers  and  enlisted  personnel 
branches.  The  JADS  Army  personnel  NCO  initially  addressed  this  with  personnel  at  OSD.  OSD 
addressed  this  with  the  different  branches  (ATEC  for  the  Army)  and  resolved  the  issue  by 
stabilizing  all  Army  personnel  for  three  years.  An  individual’s  records  were  subsequently  coded 
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at  the  local  personnel  servicing  center  (after  prodding  from  the  JADS  personnel  NCO)  to  reflect 
this  stabilization. 

The  justification  for  the  tour  stabilization  of  Army  personnel  assigned  to  JADS  JTF  was  based  on 
Army  Regulation  614-5,  Stabilization  of  Tours,  Table  2-1,  Stabilized  Tour  Lengths  of 
Organizations,  that  lists  offices  of  the  Secretary  of  Defense  (1B3AA)  as  a  36-month  tour  for 
commissioned  officers,  warrant  officers  and  enlisted  personnel. 

Army  enlisted  positions  were  initially  not  coded  for  a  three  year  stabilization  which  resulted  in 
two  NCOs  receiving  change  of  station  orders  within  12  months  of  assignment  to  JADS.  The 
JADS  Army  personnel  NCO  worked  through  Fort  Bliss,  Texas,  personnel  management  and 
ATEC  to  get  all  enlisted  positions  coded  as  stabilized  for  three  years  which  greatly  reduced 
turnover  and  turmoil. 

•  Someone  on  the  JTF  support  team  should  be  responsible  for  coordinating  Army  in¬ 
processing 

All  Army  personnel  assigned  to  the  JADS  JTF  were  assigned  to  Fort  Bliss,  Texas  for 
administrative  purposes  and  processed  through  that  installation  prior  to  arrival.  Because  of  this, 
it  was  often  difficult  to  keep  personnel  and  financial  records  accurate  and  up  to  date.  Many  hours 
were  spent  on  the  phone  trying  to  rectify  errors  in  records  and  stabilize  pay  and  allowances. 

Army  military  members  should  maintain  close  contact  with  their  assigned  JTF  support  team 
while  in-processing.  This  normally  is  the  Army  installation  located  closest  to  the  JTF.  This 
measure  will  ensure  all  required  personnel  and  financial  actions  are  taken. 

4.6.3  Air  Force  Personnel 

•  Coordinate  with  the  Air  Force  Personnel  Center  (AFPC)  for  stabilization  or  operational 
deferments  of  Air  Force  personnel. 

The  director  prepared  written  justification  to  stabilize  time  on  station  of  Air  Force  personnel  for  a 
period  ranging  from  three  to  five  years.  The  number  of  years  was  determined  by  the  projected 
needs  of  the  JTF.  This  justification  was  submitted  to  HQ  AFPC,  Randolph  Air  Force  Base, 
Texas,  through  the  director  of  personnel,  1 1^  Wing,  Bolling  AFB,  Washington  D.C. 

For  the  most  part,  the  justification  was  accepted,  and  the  authorization  for  the  stabilization  of 
personnel  was  published  in  The  Stabilization  Tour  Guide  (maintained  at  HQ  AFPC).  Specific 
officer  assignment  teams  voiced  exceptions  to  the  justification  to  the  extent  that  the  needs  of  the 
Air  Force  could  not  support  such  a  request.  As  a  result,  individuals  in  the  communications- 
computer  (33S3X)  and  analyst  (61  S3  A)  career  fields  were  disapproved  for  stabilized  tours  of 
duty.  Nevertheless,  there  is  an  “unwritten”  rule  that  individuals  more  than  likely  will  remain  on 
station  for  three  years.  Although  not  totally  confident  in  the  unwritten  rule,  we  had  no  recourse 
but  to  rely  on  the  system  to  protect  our  personnel  from  being  reassigned  until  after  three  years. 
This  proved  to  be  the  case. 
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As  the  expiration  for  tours  of  duty  for  certain  individuals  drew  near,  it  became  apparent  that  the 
JTF  could  ill  afford  to  lose  these  individuals  and  their  expertise.  As  a  result,  the  director  once 
again  provided  justification  and  requested  a  one-year  operational  deferment  that  would  extend 
the  original  stabilized  tour  of  duty.  In  all  cases,  the  operational  deferment  was  approved. 
Without  this  request,  individuals  would  have  been  reassigned.  Had  these  individuals  been 
reassigned  with  the  amount  of  time  remaining  before  the  JTF  completed  its  mission,  there  would 
have  been  no  replacement  personnel  identified. 

Stabilization  of  tours  of  duty  and  operational  deferments  (as  needed)  are  a  must  for  any  JTF. 

•  Air  Force  personnel’s  senior  rater  chain  should  go  through  OSD. 

When  JADS  was  established,  the  opinion  of  some  directors  and  their  deputies  was  that  if  Air 
Force  JT&E  personnel  were  rated  and  evaluated  for  promotion  within  OSD  channels  their 
promotion  opportunity  was  reduced.  Several  individuals  felt,  for  example,  if  a  communications- 
computer  officer  assigned  to  a  JTF  competed  with  an  astronaut  or  special  assistant  to  the 
President  of  the  United  States,  the  advantage  would  favor  the  latter  two.  For  this  reason,  some 
JT&Es  proposed  reporting  chains  through  their  services  rather  than  through  the  Deputy  Director, 
System  Assessment.  The  problem  was  that  this  approach  violated  Air  Force  directives. 

The  JADS  personnel  chief  conducted  a  careful  screening  of  past  promotion  board  files  and 
determined  that  officers  assigned  to  a  JTF  did  not  compete  at  the  same  management  level  as 
personnel  described  above.  Only  those  officers  under  the  umbrella  of  the  Under  Secretary  of 
Defense,  Acquisition  and  Technology  (USD  [A&T])  compete  with  one  another.  The  JADS 
senior  enlisted  person  provided  direct  support  for  several  USD  (A&T)  management-level  review 
board  proceedings  and  reported  that  officers  with  strong  records  and  well-written  promotion 
recommendations  were  the  ones  who  were  promoted. 

This  issue  continued  to  be  debated  at  various  forums,  and  each  JT&E  established  its  rating  chain 
either  through  OSD  or  service  channels  until  January  1999  when  the  Deputy  Director,  Systems 
Assessment  established  uniform  performance  evaluation  and  promotion  recommendation  policies 
for  all  JT&Es. 

4.6.4  Navy  Personnel 

•  Getting  Navy  personnel  required  direct  support  from  a  Navy  flag  officer  commanding 
an  organization  who  wanted  to  help  sponsor  the  work  being  done  by  the  JTF. 

It  is  far  more  difficult  for  the  Department  of  the  Navy  to  provide  human  resources  to  JTFs.  This 
is  due  to  their  personnel  structure.  If  a  naval  organization  provides  any  billets  to  a  JTF,  it  will 
lose  those  billets  for  the  duration  of  the  JT&E.  As  previously  indicated,  COMOPTEVFOR 
provided  two  military  members  to  JADS  JTF.  However,  there  were  no  permanent  allocations  of 
personnel.  Rather,  the  joint  test  director  working  directly  with  COMOPTEVOR  negotiated  the 
release  of  two  personnel  to  provide  support  to  JADS.  Under  these  conditions,  the  required  skills 


25 


to  fill  a  JTF  position  may  or  may  not  be  available.  About  a  year  after  JADS  lost  its  Navy  deputy 
and  analyst,  the  JTF  director  negotiated  for  and  received  a  Navy  deputy  (civilian)  with  a  T&E 
and  JTF  background.  Usually  Navy  facilities,  which  stand  to  benefit  most  from  the  JTFs 
out(uome,  are  willing  to  provide  some  of  the  human  resources.  Negotiation  with  these 
organizations  should  start  early  in  the  JFS  phase  of  the  program. 

4.6.5  Professional  Development 

•  Recommend  the  JT&E  program  manager  establish  a  process  for  JT&E  acquisition 
personnel  to  receive  credit  for  test  and  evaluation  activities  and  to  compete  for 
acquisition  training  slots  through  OSD  channels. 

Acquisition  training  is  required  for  personnel  in  various  career  fields  and  specifically  acquisition 
career  fields  to  meet  the  requirements  of  the  Defense  Acquisition  Workforce  Improvement  Act 
(DAWIA).  There  are  three  levels  of  certification  for  eleven  career  functional  areas.  Each  level 
identifies  minimum  requirements  in  education,  experience  (time  served  in  acquisition  duties), 
and  formal  training  and  builds  upon  the  previous  level.  It  is  imperative  that  acquisition  personnel 
receive  the  training  necessary  for  their  appropriate  career  level  for  them  to  advance  in  their  career 
field.  In  many  cases,  their  certification  levels  directly  impact  their  follow-on  assignments. 

For  various  reasons,  JT&E  manpower  positions  are  not  identified  as  acquisition  positions  even 
though  many  of  the  positions  are  from  acquisition  career  fields  and  the  JT&E  program  is 
managed  under  USD  (A&T).  Therefore,  JT&E  personnel  from  acquisition  career  fields  do  not 
automatically  get  credit  for  time  served  in  a  test  and  evaluation  organization,  and  their  formal 
training  is  not  actively  managed  nor  sponsored  by  any  organization. 

Individuals,  who  were  motivated  to  advance  in  their  career  field,  researched  course  availability 
on  the  Internet,  arranged  with  the  course  sponsors  for  vacant  slots,  and  convinced  the  JADS 
director  to  sponsor  them  for  the  course  by  paying  for  travel  expenses.  JADS  recommends  that 
the  JT&E  program  manager  establish  a  process  for  acquisition  personnel  to  receive  credit  for  test 
and  evaluation  activities  and  to  compete  for  acquisition  training  slots  through  OSD  channels. 

Until  formal  management  is  accomplished,  JT&E  directors  should  to  be  sensitive  to  this 
requirement,  help  individuals  with  their  professional  development  and  ensure  that  individuals  get 
the  training  they  need  for  advancement.  This  is  something  that  needs  to  be  followed  closely  by 
the  JT&E  to  ensure  that  it  doesn’t  “fall  through  the  cracks.” 

•  Positions  in  Air  Force  Advanced  Conununications-Computer  Systems  Officer  Training 
(ACOT)  are  not  automatically  assigned  to  officers  assigned  to  a  JT&E. 

ACOT  provides  mid-career  training  for  the  communications  and  information  staff  officer  (AFSC 
33S4).  To  be  eligible  to  attend,  officers  must  be  in  the  grade  of  captain  through  lieutenant 
colonel  with  at  least  four  years  of  field  experience  in  the  communications  and  information 
specialties.  Additionally,  officers  must  have  at  least  eight,  but  no  more  than  13,  years  of 
commissioned  service.  Currently  the  course  length  is  seven  weeks  and  three  days.  ACOT  is 
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necessary  formal  training  for  a  career  33S4  officer.  Since  slots  are  given  to  major  commands 
(MAJCOMs)  and  JADS  JTF  reports  directly  to  OSD,  this  can  and  has  caused  problems  for 
personnel  wishing  to  attend  the  course. 

During  JADS  tenure,  we  had  one  officer  who  met  the  criteria  to  attend  ACOT.  Unfortunately, 
she  had  to  work  her  own  slot.  She  called  AFPC  directly  and  through  hard  work  and  perseverance 
was  able  to  obtain  a  slot  enroute  to  her  next  assignment.  This  is  not  the  optimal  way  to  work 
slots. 

The  optimal  way  to  work  ACOT  slots  is  to  work  with  the  11^'’  Wing  at  Washington  D.C.  It 
provides  administrative  support  for  such  issues  since  JTFs  do  not  have  a  host  MAJCOM.  The 
1 1'**  Wing  reviews  communications  officers’  records  for  eligibility  and  sends  requests  to  units  for 
nominees.  Nominees  meet  a  board  at  the  1 1“’  Wing,  and  it  is  then  determined  who  will  get  a  slot 
as  primary  or  alternate.  Once  selected,  officers  are  given  a  class  date. 

4.7  Budget 

•  One  of  the  most  critical  positions  on  the  entire  test  force  is  that  of  the  person  who 
manages  both  the  budget  and  the  resources,  but  that  cannot  be  the  only  person  who  is 
responsible  for  developing  and  executing  a  budget. 

JADS  was  blessed  to  have  a  senior  AF  NCO  who  was  very  capable  of  managing  the  myriad  of 
procedures  and  sources  of  money  that  every  joint  test  will  face.  This  budget/resource  manager,  if 
competent,  needs  to  have  the  authority  to  interface  with  all  members  of  the  joint  test  force  who 
have  budgetary  responsibility.  In  JADS,  the  team  leads  controlled  their  own  budgets.  They  were 
given  targets  for  test  costs,  travel  and  training  by  fiscal  years  (FY)  and  an  overall  budget  for  all 
years.  This  allowed  them  to  control  costs  and  have  the  flexibility  to  prioritize  their  requirements 
and  cover  overages  within  their  budget  without  having  to  request  additional  funding  from  the 
JTD.  This  put  the  responsibility  for  test  costs  in  the  team  lead’s  control.  This  allowed  the  team 
leads  to  negotiate  cost  and  content  to  fit  into  their  budgets  and  to  find  indirect  or  direct  funding  to 
cover  the  shortfalls. 

•  Be  prepared  to  adjust  the  JT&E  budget  as  circumstances  surrounding  your  program 
change. 

OSD  provided  the  primary  direct  funding  for  the  JADS  JTF.  The  original  proposed  budgets: 

JADS  $18,043,000 

JADS  EW  Test  $  6,725,000 

Total  actual  OSD  funding: 

JADS  $20,503,000 

JADS  EW  Test  $  8,165,000 
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The  increases  of  $3.9  million  in  the  budgets  were  for  the  SIT  delays,  additional  effort  for  SIT  and 
infrastructure  costs  in  FY99  and  FYOO.  The  delays  in  the  ETE  Test  made  the  FY99  completion 
date  for  JADS  unlikely,  which  caused  an  increase  for  infrastructure  costs  for  FY99  and  FYOO. 

Tables  3  and  4  show  the  direct  and  indirect  funding  received  by  each  source.  The  direct  funds 
include  funds  received  by  Military  Interdepartmental  Purchase  Request  (MIPR)  or  budget 
allotments  directly  to  JADS.  Direct  funding  by  the  lead  service  included  civilian  salaries  and 
training,  maintenance,  communication.  General  Services  Administration  (GSA)  vehicle,  and 
facility  lease  cost  because  Kirtland  AFB  could  not  provide  office  space  in  FY96.  Indirect  cost 
was  the  military  personnel  salaries. 


Table  3.  Direct  Funding  Profile 


FY  94 

FY  95 

FY96 

FY97 

FY98 

FY99 

FYOO 

TOTAL 

OSD 

$477,905 

$3,257,000 

$6,330,000 

$8,100,000 

$5,430,000 

$4,219,000 

$ 

832,000 

$28,645,905 

CTEIP 

0.00 

$  650,000 

$  500,000 

$1,100,000 

$  400,000 

0.00 

0.00 

$  2,650,000 

CROSSBOW 

0.00 

$  596,827 

$  164,919 

0.00 

0.00 

0.00 

0.00 

$  761 ,746 

AIR  FORCE 

$  22,900 

$  240,572 

$  425,283 

$  416,716 

$  452,813 

$  506,500 

$ 

230,800 

$  2,295,584 

ARMY 

0.00 

0.00 

$  78,700 

$  20,000 

$  20,000 

0.00 

0.00 

$  118,700 

TOTAL 

$500,805 

$4,744,399 

$7,498,902 

$9,636,716 

$6,302,813 

$4,725,500 

$ 

1 ,062,800 

$34,471,935 

Table  4.  Indirect  Funding  Profile 


FY  94 

FY  95 

FY96 

FY97 

FY98 

FY99 

FYOO 

TOTAL 

AIR  FORCE 

ARMY 

NAVY 

$50,052 

0 

0 

$730,772 

$208,841 

$41,454 

$1,228,542 

$679,449 

$159,612 

$1,235,876 

$802,993 

$156,751 

$1 ,389,704 
$1 ,056,461 

$51,520 

$1,335,651 

$857,566 

0 

$455,797 

$129,320 

0 

$6,426,394 

$3,734,630 

$409,337 

TOTAL 

$50,052 

$981,067 

$2,067,603 

$2,195,620 

$2,497,685 

$2,193,217 

$585,117 

$10,570,361 

4.8  Facilities 

•  Leasing  off-base  facilities  is  an  option  if  adequate  government  facilities  are  not 
available. 

Facility  costs  are  funded  with  lead  service  operation  and  maintenance  (O&M)  dollars  (one-year 
money).  The  lead  service  pays  all  costs  to  house  the  JTF:  communications,  utilities,  lease  and 
maintenance  costs  on  administration  machines  and  base  support  costs.  If  the  lead  service  is 
unable  to  find  a  place  for  you  to  work,  leasing  is  an  option.  The  lead  service  pays  all  costs  for  the 
leased  facility.  The  cost  can  be  obligated/expensed  by  the  terms  of  the  lease  not  by  fiscal  year  as 
stated  in  Public  Law  2410a,  Appropriated  Funds  Availability  for  Certain  Contracts  for  12 
Months.  It  states  that  “  Funds  appropriated  to  the  Department  of  Defense  for  a  fiscal  year  shall 
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be  available  for  payments  under  contracts  for  any  of  the  following  purposes  for  11  months 
beginning  at  any  time  during  the  fiscal  year....  (2)  The  lease  of  real  or  personal  property, 
including  the  maintenance  of  such  property  when  contracted  for  as  part  of  the  lease  agreement." 
This  means  that  if  your  lease  is  an  August-July  term,  you  can  fund  the  whole  first  year  using  the 
FY  money  current  for  August. 

Sources  for  getting  a  leased  facility  are  GSA  and  the  Army  Corps  of  Engineers.  GSA  takes  a  few 
years  to  get  the  process  done.  JADS  was  able  to  obtain  support  from  the  Corps  of  Engineers  in 
Baltimore.  They  did  all  the  paperwork  needed  to  get  the  facility  when  needed. 

Their  address  is: 

Department  of  the  Army 

Baltimore  District 

U.  S.  Army  Corps  of  Engineers 

P.O.  Box  1715 

Baltimore  MD  21203-1715 

The  point  of  contact  (POC)  is  D.R.  Middleton,  (410)  854-7039. 

4.9  Supply  Support 

•  Have  two  avenues  to  choose  from,  the  International  Merchant  Purchase  Authorization 
Card  (IMPAC)  and  a  supply  account,  when  purchasing  supplies. 

When  the  determination  was  made  that  supply  support  would  be  necessary  to  the  daily  operation 
of  JADS,  certain  records  needed  to  be  created  through  the  local  base  supply  organization. 
Because  Kirtland  AFB  is  the  designated  host  for  JADS  JTF,  Air  Force  supply  manuals  applied  to 
the  type  of  support  received  from  base  supply. 

AFMAN  23-110,  USAF  Supply  Manual,  Volume  VU,  Part  13  outlines  procedures  for 
establishing  organizational  records  and  supply  accounts.  Base  supply  Records  Maintenance 
element  is  the  point  of  contact  for  establishing  an  account.  A  KAFB  Form  219,  Establishment  of 
Organization  Records,  is  used  to  create  and  record  the  organizational  record.  Records 
Maintenance  maintains  this  form. 

The  creation  of  this  record  enables  the  customer  to  order  from  base  supply.  It  enables  supply  to 
process  listings  for  the  organization  to  maintain  their  account.  The  record  contains  indicative 
data  on  the  type  of  unit  and  its  mission.  It  also  reflects  priority  of  support.  This  record  is 
identified  by  a  three-digit  numerical  code  and  is  linked  to  the  unit.  A  Project  Funds  Management 
Record  (PFMR)  allows  ordered  items  to  be  paid  for  at  the  time  of  purchase. 

The  IMPAC  card  and  the  supply  account  combined  gave  the  unit  two  avenues  for  acquiring 
resources,  gave  JADS  maximum  flexibility  and  enabled  continuous  operation.  When  parts  or 
supplies  were  not  available  in  base  supply,  the  IMPAC  card  gave  JADS  the  ability  to  purchase 
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those  same  items  commercially  on  the  local  economy  or  from  a  catalog.  This  option  saved 
enormous  amounts  of  time  that  would  have  been  spent  waiting  for  back-ordered  items  from  base 
supply. 

•  Appoint  an  equipment  custodian  to  keep  track  of  your  equipment. 

To  support  daily  operations,  certain  equipment  items  were  required.  Equipment  items  are 
defined  in  AFMAN  23-1 10  Volume  H,  Part  13,  Chapter  8  as  “items  of  durable  nature  ca,pable  of 
continuing  repetitive  use.  Items  other  than  supplies  needed  to  outfit  a  unit  or  individual.  These 
are  also  known  as  nonexpendable  items  and  are  required  to  be  maintained  on  records  for 
accountability.  Equipment  items  are  ordered  on  an  AF  Form  601,  Equipment  Action  Request, 
and  are  approved  at  levels  identified  in  the  table  of  allowances.  Air  Force  Materiel  Command 
(AFMC)  determines  which  items  are  to  be  coded  as  equipment  items. 

JADS  established  an  equipment  account  for  equipment  accountability  in  accordance  with 
AFMAN  23-110,  Volume  VH,  Part  13,  Chapter  8.  The  equipment  account  was  established 
through  the  Equipment  Management  element  at  base  supply.  The  organizational  record 
(mentioned  above)  must  be  created  prior  to  establishing  an  equipment  account.  The  JADS 
director  appointed  a  custodian  to  manage  all  the  equipment  for  the  unit.  Equipment  custodian 
responsibilities  are  listed  in  AFMAN  23-1 10,  Volume  VII,  Part  13,  Chapter  8. 


4.10  Security 

•  Building  security  in  an  off-base  leased  facility  takes  careful  planning. 

Lease  Agreement.  The  lease  agreement  with  the  building  owners  required  securing  the  leased 
facility  and  premise.  The  following  paragraph  was  part  of  the  lease  agreement:  “The  government 
shall  have  the  right  to  secure  the  leased  premise  with  the  lessor’s  approval,  which  shall  not  be 
unreasonably  withheld.” 

Off-Base  Facility  Security  Survey.  A  security  survey  of  the  building  was  conducted.  We  used 
DoD  references  to  determine  security  requirements. 

•  DoD  Regulation  5200.8,  Physical  Security  Program 

•  DoD  Regulation  5200. 1 ,  Information  Security  Program 

•  Military  Handbook- 10 13/1  A,  Design  Guidelines  for  Physical  Security  of  Facilities 

Physical  Security  Assessment.  Physical  security,  according  to  DoD  Regulation  5200.8, 
involves  the  safeguarding  of  personnel;  preventing  unauthorized  access  to  the  facility, 
equipment,  material,  and  documents;  and  safeguarding  them  against  espionage,  sabotage, 
damage,  and  theft. 

The  type  and  scope  of  the  functions  housed  in  the  off-base  facility  will  determine  the  threat  level. 
Our  facility  was  determined  to  be  a  threat  type  low  (individual(s)  or  insider(s)  working  alone  or 
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in  a  small  group).  Threat  examples:  casual  intruders;  pilfers  and  thieves;  overt  intelligence 
collectors;  passive  demonstrators.  Pilferable  items  included  supplies,  computers,  typewriters, 
merchandise,  and  equipment.  The  intruder’s  goal  is  likely  to  steal  or  destroy  assets,  compromise 
information,  or  disrupt  operations.  Vandals,  activists,  extremist  protesters  and  terrorists  usually 
target  buildings  rather  than  the  assets  housed  in  the  buildings. 

Under  a  low-level  threat  scenario,  it  is  not  considered  practical  to  attack  walls  using  only  a 
limited  set  of  hand-held  tools.  Such  a  threat  would  likely  attack  the  doors,  windows,  or  other 
more  vulnerable  part  of  the  facility.  Likewise,  it  is  not  practical  to  attack  roofs  or  floors. 
Consequently  any  permanent  roof  or  floor  construction  can  be  considered  adequate  for  a  low- 
level  threat.  Modifying  door  and  windows,  considered  the  most  vulnerable  points,  to  increase  the 
protection  level  would  be  costly. 

Being  in  an  off-base  leased  facility  does  not  offer  any  security  in  depth  or  a  buffer  zone  normally 
expected  from  the  secure  environment  on  a  military  installation.  As  a  result,  it  was  prudent  to 
lock  building  entrances  on  a  daily  basis. 

Areas  focused  on  during  the  building  assessment  included  exterior  security  lighting;  building 
entry/exit;  windows;  vehicle  parking;  open  storage  of  classified  information;  classified 
meeting/conference  area;  telephone  control  room;  use  of  existing  vaults;  electric  control/breaker 
box;  TEMPEST  (special  shielding  against  electromagnetic  radiation)  considerations;  and 
publishing  operating  instructions  for  the  facility. 

Alarm  Installation  by  an  Industrial/Commercial  Alarm  Company.  The  building  survey 
concluded  that  the  following  security  intrusion  detection  devices  should  be  installed; 

•  Door  panic  hardware 

•  Cipher  lock  (2  main  entrance  doors) 

•  Glass  breakage  sensors  for  all  windows 

•  Interior  building  sensors 

•  Separate  alarm  partition  for  the  Test  Control  and  Analysis  Center  (TCAC)  (secure  room) 

The  type  of  alarm  that  was  installed  required  is.suing  access  codes  to  each  individual.  Personnel 
were  trained  on  building  code  access  and  alarm  procedures. 

Because  we  were  located  off  base  and  would  not  receive  base  law  enforcement  assistance,  a 
civilian  security  response  company  was  hired  to  respond  within  15  minutes  of  an  alarm 
activation. 

As  an  added  precaution  for  the  protection  of  classified  material,  an  alarm  response  team  was 
established  to  respond  in  conjunction  with  the  civilian  security  response  company.  The  alarm 
company  was  provided  with  a  list  of  names  of  individuals  able  to  respond  swiftly  to  alarm 
activation. 
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A  lesson  learned  was  that  careful,  documented  communication  with  the  alarm  company  is 
essential.  Fax  receipts  or  alarm  company  verification  faxes  are  needed  to  confirm  that  all 
pertinent  changes  to  the  facility  alarm  system  are  received. 

•  Maintaining  and  updating  Air  Force  personnel  security  clearances  should  be  done 
through  the  base  security  office. 

Periodic  reinvestigations  ot  Air  Force  members  not*read  into  a  SAP  were  coordinated  through 
the  base  security  forces  Personnel  Security  Office.  The  Air  Force  Automated  Security  Clearance 
Approval  System  (AS CAS)  listing  was  used  to  maintain  up-to-date  clearance  information  on  AF 
personnel.  The  ASCAS  listing  has  now  been  replaced  by  the  Sentinel  Key  Automated  Database. 
The  base  security  police  also  provided  training  and  shared  information  on  DoD-wide  changes  in 
security  procedures. 

•  Maintaining  and  updating  Army  personnel  security  clearances  should  be  done  through 
the  servicing  PSC. 

Periodic  reinvestigations  of  Army  personnel  were  coordinated  through  the  servicing  personnel 
services  center  (PSC).  The  PSC  identified  soldiers  needing  periodic  updates  during  annual  birth 
month  or  in  processing  record  audits  and  notified  them  of  the  requirement  to  update  theii 
clearance.  Defense  Investigative  Service  conducted  background  investigations  as  required. 

•  Contractor  security  clearances  required  certain  specific  information  to  be  on  file  with 
the  JTF. 

Initial  contractor  security  clearances  were  faxed  to  the  JADS  security  manager  by  the  contractor- 
assigned  organization  security  manager.  All  contractor  security  clearances  required  a  contract 
numl)er  as  well  as  full  name,  title,  citizenship,  social  security  number,  type  of  clearance,  date 
clearance  granted,  date  of  birth,  place  of  birth,  length  of  stay  for  work  requirement  (up  to  one 
year),  and  the  facility  identification  (ID)  of  who  granted  the  clearance.  For  contractors,  in  most 
cases,  the  facility  that  grants  the  clearance  is  the  Defense  Security  Services  Operations  Center 
(DSSOC).  A  current  list  with  proper  clearance  and  identification  of  all  JADS  military  personnel 
and  contractor  personnel  was  kept  readily  on  file. 

•  Document  control  required  close  interaction  among  the  security  manager,  team  leads 
and  legacy  personnel. 

4.10.1  Classification  of  Documents 

The  JTD  was  specifically  delegated  in  writing  as  the  organization’s  original  classification 
authority  (OCA).  Technical  documents/reports  (original  or  derivative)  created  by  JADS  were 
reviewed  thoroughly  by  a  check  system  designed  to  comply  with  Executive  Order  12958, 
Classified  National  Security  Information.  Executive  Order  12958  requires  that  classified 
national  security  information  be  marked  to  alert  holders  and  recipients  of  classified  information, 
to  identify  the  exact  information  or  portion  that  needs  protection,  to  provide  guidance  for 
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downgrading  and  declassification  of  classified  information,  to  give  the  reason  for  the  initial 
classification  decision,  and  to  warn  the  holders  of  any  special  access,  controls,  or  safeguarding 
requirement.  DoD  Regulation  5200.1  and  DoD  5200.1 -PH,  DoD  Guide  to  Marking  Classified 
Documents,  were  used  to  adhere  to  Executive  Order  12958. 

4.10.2  Distribution  Statements 

Technical  documents  created  by  JADS  were  reviewed  under  a  check  system  for  proper 
assignment  of  distribution  statements.  This  check  system  complied  with  DoD  Directive  5230.24, 
Distribution  Statements  on  Technical  Documents,  and  required  that  distribution  statements  be 
placed  on  all  technical  documents  both  classified  and  unclassified.  The  purpose  of  assigning 
distribution  statements  was  to  facilitate  control  of  classified  and  unclassified  documents  for 
distribution  release,  dissemination  without  additional  approvals  or  authorizations,  and  to 
eliminate  the  need  to  repeatedly  refer  questions  to  the  originating  activity.  Ensure  team  leads  are 
briefed  on  the  responsibility  of  the  OCA,  Executive  Order  12958,  and  DoD  Directive  5230.24. 

4.10.3  Internet  Publication  Security  Procedures 

An  Internet  web  site  was  established  to  provide  a  place  for  the  public  to  view  and  retrieve  JADS 
information.  All  information  on  the  site  was  reviewed  under  our  check  system  for  proper 
classification  and  assignment  of  distribution  statements.  Only  those  documents  determined  to  be 
distribution  A  (approved  for  public  release;  distribution  unlimited)  were  posted  on  the  web  site. 
Make  authors  of  technical  documents  aware  of  the  plan  to  put  documents  onto  a  web  site.  The 
writer  can  then  focus  on  preparing  the  document  for  unlimited  distribution. 

•  The  JADS  JTF  did  significant  work  in  the  area  of  security  administration.  Annex  A 
provides  information  appropriate  for  JTF  personnel  responsible  for  all  areas  of 
security  administration. 

4.11  Test  Management 

•  While  the  outline  test  plan  (OTP)  was  critical  to  obtaining  Army  resources  to  support 
the  JTF,  the  test  resource  plan  (TRP)  was  less  critical  for  Air  Force  support  and  the 
consolidated  resource  estimate  (CRE)  provided  utility  only  during  the  JFS  and  initial 
JTF  stages. 

The  CRE  was  put  together  during  the  feasibility  study  and  updated  irregularly  during  the  first 
year  and  a  half  of  the  test.  Once  detailed  planning  and  coordination  of  the  tests  was  underway, 
updating  the  CRE  became  more  difficult  and  provided  little  value  to  our  test  activities.  The  TRP 
was  important  in  identifying  Air  Force  resources  required  to  man  the  JT&E  but  coordination  for 
actual  test  support  required  a  program  introduction  document  (PID)/statement  of  capability 
(SOC)  process  with  each  Air  Force  participant.  Most  Air  Force  support  could  have  been  and  was 
accomplished  with  a  PID/SOC  process  and  no  TRP  input.  Army  support  however,  required  a 
detailed  input  into  the  OTP  process.  The  OTP  request  was  staffed  through  ATEC  DCSOPS  who 
worked  the  approval  with  the  test  schedule  and  review  committee  (TSARC).  Once  support  was 
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approved  by  the  TSARC,  orders  were  passed  through  the  various  MAJCOMS  to  the  units 
designated  to  support  JADS  testing.  Changes  in  test  support  required  updates  to  the  OTP 
submission.  Certain  Army  elements  operate  under  a  reimbursable  basis,  and  a  process  similar  to 
PID/SOC  was  used  to  contract  for  their  support.  For  both  the  TRP  and  OTP,  a  major  from  each 
service  was  made  responsible  for  submission  of  the  requests.  The  service  deputies  reviewed  the 
plans  prior  to  submission  to  the  director  for  signature  and  forwarding  to  the  respective  service. 

•  Complex  JTFs  with  overlapping  test  phases  may  require  both  an  operations  calendar 
and  an  administrative  calendar, 

JADS  tried  a  number  of  methods  for  maintaining  a  useful  calendar.  Initial  attempts  to  have  the 
director’s  secretary  keep  and  update  an  administrative  calendar  failed  because  of  the  skill  level  of 
the  secretary  and  the  fluid  nature  of  test  activities.  After  investigating  several  software 
alternatives,  the  JADS  executive  officer  developed  an  Excel  spreadsheet  calendar  that  was 
available  to  all  JADS  personnel  on  the  server.  Team  leads  were  responsible  for  updating  the 
calendar  in  their  areas  of  responsibility.  The  calendar  was  reviewed  at  each  weekly  staff 
meeting. 


As  test  activities  increased  in  complexity  and  number,  JADS  needed  to  develop  a  mechanism  for 
deconflicting  internal  resources  among  competing  tests.  The  solution  was  an  extension  of  the 
Excel  administration  calendar.  A  second  calendar,  the  operations  calendar,  was  added  to  the 
spreadsheet  and  managed  by  the  Program  Control/Support  team  lead.  At  monthly  team  meetings 
resources  were  scheduled  in  detail  30  days  out  and  estimated  90  days  out.  This  process  allowed 
test  team  leads  to  shift  critical  activities  to  ensure  adequate  internal  resources  were  available  for 
their  tests  and  gave  the  matrix-supported  team  leads  visibility  of  upcoming  requirements. 

•  Effective  communication  was  a  key  ingredient  in  JADS’  successes  in  testing,  in  the  conduct 
of  its  overall  mission,  its  efficiency  and  high  morale. 

JADS  made  a  conscious  effort  to  ensure  open  and  effective  communications  both  vertically  and 
horizontally  within  the  organization.  The  director  conducted  weekly  staff  meetings  to  identify 
and  resolve  issues,  and  to  provide  cross  communication  to  other  leadership  team  members 
regarding  plans  and  activities  in  areas  of  the  JTF.  Additionally,  the  director  conducted  monthly 
program  reviews,  attended  by  the  leadership  team,  at  which  each  functional  area  within  the  JTF 
was  scrutinized.  Finally,  the  director  held  monthly  Director  Calls,  attended  by  all  JTF  personnel, 
both  government  and  contractor.  Information  and/or  training  germane  to  the  whole  group  was 
conducted,  recaps  of  activities  by  team  leads  provided,  awards  and  decorations  administered,  etc. 
The  end  result  was  a  well-informed,  cohesive  and  efficient  JTF  with  high  morale. 

•  Leadership  team  off  sites  are  critical  to  keeping  the  big  picture  focused  and  maintaining 
high  organizational  morale. 


'  The  Membership  Team  was  comprised  of  the  Director,  technical  advisor,  Service  deputies,  team  leads,  contractor 
program  manager,  and  secretaries  (contractor  and  government). 
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The  leadership  team  consisted  of  the  JTD,  technical  advisor,  service  deputies,  the  SAIC  program 
manager,  team  leads,  senior  enlisted  advisor  and  secretaries.  Their  primary  responsibility  as 
members  was  to  assist  the  director  in  keeping  the  JT&E  focused  on  the  critical  areas  of  the 
mission  and  to  ensure  that  JADS  was  an  efficient  and  enjoyable  organization.  Each  September, 
the  director  gathered  the  leadership  team  at  an  off-site  location  to  develop  goals  for  the  upcoming 
fiscal  year.  Two  sets  of  goals  were  developed:  JADS  goals  relating  to  program  execution  and 
leadership  team  goals  relating  to  creating  an  efficient  and  enjoyable  organization.  Once  goals 
were  established,  the  director  briefed  them  to  all  JADS  personnel,  and  quarterly  on-site/off-site 
meetings  were  held  to  review  the  status  of  achieving  each  goal. 

4.12  Program  Advocacy 

•  A  significant  amount  of  the  JTD’s  time  was  required  for  program  advocacy  briefings 
and  program  reviews. 

A  necessary  task  and  a  major  demand  on  the  test  director's  time  and  talent  are  the  briefings 
required  to  get  the  program  chartered,  then  to  keep  the  program  alive.  Of  equal  importance  are 
program  reviews  and  updates  to  the  various  oversight  functions  and  securing  support  from  the 
many  organizations  whose  voluntary  services  are  vital  to  test  execution.  In  an  attempt  to  provide 
future  test  directors  with  an  idea  of  the  magnitude  of  the  program  advocacy  demands,  the  major 
briefings,  reports,  and  TDYs  for  program  advocacy  from  initial  feasibility  study  through  the  final 
test  report  are  listed. 

Program  Reviews  and  Updates.  This  category  covers  all  the  post-charter  trips  and  briefings 
required  to  maintain  overall  JADS  JT&E  support  and  funding  from  OSD  and  the  services.  Each 
trip  lasted  an  average  of  two  working  days  and  involved  the  JTD  and  one  or  more  members  of 
the  JADS  staff  (e.g.,  technical  advisor,  service  deputy,  principle  investigator).  A  formal  briefing 
was  given  on  almost  all  trips.  The  briefings  on  the  norm  were  forty-five  minutes  with  fifteen 
minutes  allocated  for  questions  and  answers.  The  shortest  briefing  lasted  fifteen  minutes  and  the 
longest  was  close  to  eight  hours.  A  significant  amount  of  staff  time  was  required  to  prepare, 
review,  and  rehearse  for  each  briefing.  Reasonable  estimates  are  as  follows:  for  each  hour  of 
presentation,  it  took  four  hours  to  prepare  a  draft,  five  hours  for  staff  review,  three  hours  of 
graphics  support,  and  two  hours  to  rehearse  and  refine  the  presentation.  So  on  the  average,  we 
invested  fourteen  hours  in  preparing  for  a  briefing  expected  to  last  for  one  hour. 

Technical  Advisory  Board  (TAB)/SAC  Briefings.  The  purpose  of  these  visits  was  to  update 
the  TAB/SAC  on  JADS  progress,  obtain  guidance  on  test  issues,  maintain  program  support,  and 
provide  interim  results.  Additionally,  JADS  was  directed  to  investigate  adding  an  additional  test, 
the  EW  test,  to  our  tasking.  This  potential  change  in  charter  required  additional  interaction  with 
the  TAB/SAC.  During  the  approximately  five-year  life  of  JADS  (including  the  JFS),  we 
conducted  ten  TAB  and  five  SAC  briefings. 

OSD  Briefings/Reports.  The  purpose  of  most  of  these  trips  was  to  discuss  administrative  issues 
such  as  funding,  manning,  and  schedule,  although  a  few  trips  were  devoted  to  technical  issues. 
The  JTD  and  several  staff  members  made  ten  trips.  The  briefings  associated  with  these  trips 
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generally  lasted  two  to  three  hours  and  required  thirty  to  forty  staff-hours  in  preparation  time.  In 
addition  to  these  trips  there  were  occasional  on-site  reviews  conducted  by  OSD  that  normally 
lasted  about  eight  hours.  These  annual  reviews  were  considered  critical  to  the  health  of  the 
program,  so  considerable  staff  time  was  invested  in  preparation.  A  conservative  estimate  would 
be  two  hundred  man-hours  including  administrative  support. 

In  addition  to  briefings,  monthly  and  annual  reports  were  sent  to  OSD.  Each  monthly  report  was 
staffed  internally  to  JADS  and  required  approximately  five  staff-hours  to  prepare  and  review. 
The  annual  report  required  approximately  thirty  staff-hours  per  year. 

Test  Support  Briefings.  These  trips  were  required  to  obtain  and  maintain  service/organization 
support  for  the  individual  tests.  These  trips  normally  involved  the  JTD,  the  team  lead,  and  a 
service  deputy.  The  briefings  associated  with  these  trips  usually  lasted  from  one  to  three  hours. 
Five  trips  were  required  for  the  SIT,  thirteen  for  the  ETE  Test,  and  four  for  the  EW  Test. 

Legacy.  The  JTD  made  at  least  forty-eight  trips  to  support  the  legacy  effort. 

Summary.  The  JTD  traveled  an  average  of  40  times  per  year  for  meetings  and/or  briefings. 
Don’t  underestimate  the  amount  of  travel,  or  the  amount  of  support  required  by  other  JT&E 
personnel  for  these  trips. 

4.13  Reporting/Legacy 

There  are  several  aspects  of  the  JADS  legacy  program  that  merit  some  discussion.  Key  elements 
of  the  legacy  program  included  a  newsletter,  a  trade  show  display  (booth),  web  site,  brochures, 
multimedia  presentation  of  reports  and  findings,  videos  and  a  professional  development  course. 
Significant  lessons  learned  are  listed  below.  Details  on  the  execution  of  each  of  the  aspects  of 
legacy  management  are  presented  in  Annex  B. 

•  Structure  your  legacy  program  to  your  specific  customer  base. 

In  JADS’  case,  our  primary  customer  base  was  the  test  and  evaluation  and  acquisition 
communities  from  all  the  services,  OSD  and  the  civilian  sector.  Consequently,  our  program  had 
a  more  marketing  kind  of  feel  than  a  JT&E  whose  customer  base  centered  on  a  specific  unified 
command  or  type  of  acquisition  program.  However,  the  basics  discussed  here  are  germane  for 
any  type  of  JT&E.  JADS  had  a  consistent,  high  level  of  attendance  at  the  professional  meetings 
of  the  T&E  and  modeling  and  simulation  (M&S)  communities.  The  JADS  web  site  allowed 
broad  access  to  information  developed  during  testing.  Because  of  the  nature  of  our  customers, 
JADS  worked  diligently  to  produce  reports  and  papers  that  were  unclassified  with  unlimited 
distribution. 

•  Start  early  and  dedicate  a  portion  of  your  test  force  to  full-time  legacy  management. 

Begin  your  legacy  program  early  in  the  test’s  lifetime.  Get  everyone  involved  in  defining  the 
legacy  program  and  build  consensus  and  get  buy-in  as  much  as  possible.  Define  who  in  the  test 


36 


are  going  to  do  what  for  legacy  and  hold  them  to  their  responsibilities.  Early  on,  it’s  difficult  to 
get  people  involved  in  legacy,  but  it’s  critical  to  the  program’s  success.  A  legacy  built  after  the 
fact  will  not  have  the  benefit  of  face-to-face  interaction  with  the  customer  base  over  the  years. 
People  won’t  look  for  or  care  about  a  report  from  an  unknown  organization.  Get  out  and  build 
an  identity  early  in  the  program.  Make  legacy  a  priority  from  the  beginning. 

Budget  for  legacy  and  dedicate  manpower  to  its  management.  Products  cost  money,  time,  and 
effort.  JADS  had  one  full-time  government  service  legacy  manager  and  halfway  through  the  JTF 
added  another  government  service  assistant.  The  amount  of  the  total  JT&E  budget  dedicated 
directly  to  legacy  was  approximately  1 .5  percent. 

•  Ensure  your  service  deputies  understand  that  legacy  is  an  important  aspect  of  their 
mission. 

An  initial  tendency,  reinforced  by  the  JT&E  Handbook,  is  to  place  service  deputies  in  charge  of  a 
particular  aspect  of  your  test  program  like  analysis  or  test  management  and  tasking  them  to 
ensure  that  their  service  is  properly  played  in  each  of  your  tests.  If  you  can  avoid  putting  them  in 
charge  of  day-to-day  execution  activities,  it  will  free  up  their  time  to  make  your  legacy  program 
stronger,  more  focused  and  more  effective.  Only  the  deputies  have  the  experience  to  properly 
identify  your  initial  customer  base  and  expand  it  as  your  emerging  results  and  conclusions 
dictate. 

4.14  JADS  Drawdown 

•  Develop  a  transition  plan  early. 

JADS  generated  the  first  draft  of  a  transition  plan  in  the  fall  of  1998,  eighteen  months  before 
official  close  out.  The  three  key  elements  of  the  transition  plan  were  a  personnel  departure 
spreadsheet,  an  office  equipment/fumiture  listing  and  a  capabilities  listing  (those  major  items 
developed  to  support  JTF  test  execution  that  needed  a  legacy  home). 

•  Delegate  authority  for  getting  rid  of  office  equipment/furniture  as  low  as  possible. 

JADS  supply  and  communications  support  personnel  were  charged  with  transitioning  as  much  of 
the  property  as  possible.  They  were  tasked  to  follow  the  rules  established  by  JT&E  Management 
Office  in  equipment  transfer.  Ownership  of  equipment,  particularly  computer  equipment,  was 
hard  to  establish.  Initially  computers  were  purchased  with  Air  Force  money  but  upgraded  with 
OSD  money.  JADS  allocated  ownership  of  computer  equipment  based  on  proportion  of  dollars 
spent  (i.e.,  if  Air  Force  and  OSD  money  were  equally  spent,  then  each  got  half  of  the  computer 
system).  Supply  accountability  in  relation  to  disbanding  JADS  was  a  joint  effort  between  JADS 
and  AFOTEC  personnel.  The  inventory  of  items  maintained  at  JADS  provided  a  list  of  items  for 
AFOTEC  to  review  and  disburse.  The  best  possible  way  to  attack  turning  furniture  and  other 
supply  items  back  to  AFOTEC  was  through  attrition.  As  items  become  unnecessary,  they  were 
identified  to  AFOTEC.  AFOTEC  responded  with  the  action  JADS  needed  to  take.  This  process 
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was  repeated  until  JADS  was  relieved  of  the  responsibility  of  all  assets  acquired  during  JADS 
operation. 

•  Maintain  open  communications  about  the  drawdown. 

The  transition  plan  and  specifically  the  personnel  drawdown  plan  were  placed  on  the  JADS 
Intranet  and  made  available  to  all  JADS  personnel.  The  JTD  made  it  a  point  to  discuss  the 
drawdown  at  monthly  director’s  calls  when  appropriate.  Keeping  all  aspects  of  the  drawdown  as 
public  as  possible  had  a  significant,  positive  impact  on  morale. 


5.0  Conclusions  and  Recommendations 


JADS  has  enjoyed  a  reputation  of  being  a  particularly  successful  JT&E  program.  Although  this 
will  come  across  as  over  simplified,  JADS’  success  can  be  attributed  in  a  simple  formula  - 
excellent  organization  plus  proactive  legacy  actions  equals  successful  JT&E.  There  are  a  huge 
amount  of  lessons  learned  and  recommendations  in  the  body  of  this  report.  A  few  will  be 
highlighted  here  that  you  should  go  to  school  on. 

5.1  Organization 

JADS  used  an  approach  to  JTF  organization  that  is  quite  different  from  what  the  JT&E 
Handbook  calls  for.  The  point  is  this  —  the  Handbook  is  a  guide.  However,  each  JT&E  program 
is  going  to  be  unique.  Therefore,  structure  the  organization  in  a  way  that  best  makes  sense  for 
your  particular  mission  and  working  environment.  Having  said  that,  there  are  facets  of  JADS’ 
approach  we  recommend  to  you  regardless  of  the  type  of  JT&E  you  have. 

•  The  mix  of  matrix  and  functional  area  teams  within  JADS  worked  exceptionally  well. 
On  a  related  note,  having  all  government  and  contractor  personnel  together  and 
integrated  into  teams  made  for  a  highly  cohesive  team  that  worked  together  for  the 
common  good. 

•  Having  the  JTF  under  one  single  roof  made  mission  execution  an  order  of  magnitude 
easier  than  if  we  had  been  geographically  separated. 

•  Possessing  all  support  functions  within  the  JTF  is,  in  our  opinion,  the  only  way  to  go. 
We  witnessed  innumerable  cases  of  “non-support”  from  other  agencies.  JADS  was 
always  able  to  get  the  support  we  needed  because  we  owned  the  support  functions.  In 
fact  we  found  ourselves  supporting  other  JT&Es  as  well. 

•  We  started  with  a  flat  organization,  but  inserted  a  Chief  of  Staff  when  it  became 
apparent  the  JTD  traveled  too  much  to  provide  day-to-day,  hands-on  leadership.  This 
worked  well. 

5.1  Legacy 

JADS  has  been  both  criticized  and  praised  for  its  legacy  program.  On  one  hand,  JADS  was 
praised  for  having  the  best  legacy  program  ever  seen.  Though  biased,  we  agree.  Criticism,  on 
the  other  hand,  fell  in  two  areas  —  it  cost  too  much  and  isn’t  relevant  to  other  types  of  JT&Es. 
JADS’  experience  is  that  you  can  have  a  highly  comprehensive  legacy  program  for  a  modest 
investment.  We  devoted  approximately  four  percent  of  our  workforce  (2  of  50)  and  1.5  percent 
of  our  budget  for  legacy.  As  for  relevance,  “experts”  said  that,  since  JADS  was  a  different  type 
of  JT&E  than  most,  other  JT&Es  shouldn’t  need  to  do  this  aggressive  style  of  approach  to  legacy. 
We  beg  to  differ.  Regardless  of  the  nature  of  your  JT&E,  you  have  a  user  community  you’re 
working  with,  and  both  information  and  products  you  need  to  embed  into  this  community  to 
make  lasting  positive  contributions.  Therefore,  the  basic  precepts  of  JADS’  legacy  program  hold 
true,  only  the  details  differ.  Here  are  some  key  points: 

•  Make  a  commitment  to  a  legacy  program  from  the  very  beginning  of  your  JT&E 
program.  Build  it  into  your  organizational  structure,  man  it  and  fund  it  from  Day  1 . 
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•  A  successful  legacy  program  needs  the  full  support  of  the  JT&E  leadership,  most 
especially  the  JTD  and  Deputies. 

•  Look  for  every  way  possible  to  get  your  word  out  on  a  continuing  basis  through  the 
life  cycle  of  your  program.  Interim  reports,  newsletters,  videos,  CDs,  presentations  at 
conferences  your  community  attends,  etc.  should  all  be  used. 

5.3  Program  Success 

JADS  was  highly  successful  across  the  board.  As  an  organization,  it  was  praised  for  its 
cohesiveness,  high  morale,  and  expertise  in  getting  the  job  done.  As  examples,  our  personnel, 
computer  and  finance  people  were  called  upon  to  help  many  others.  JADS  was  also  very 
successful  in  conducting  rigorous  T&E  events  which  fulfilled  JADS’  charter  and  provided  valid, 
believable  data  to  our  user  communities.  Finally,  JADS  is  making  widespread  and  long-lasting 
contributions  to  the  community  thanks  to  its  superb  legacy  program. 
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Annex  A 

Security  Issues  and  Information 

Al.  Storage  of  Classified  Material 

General  Services  Administration-approved  security  containers  were  obtained  for  proper  storage 
of  generated  or  acquired  classified  information. 

A2.  Receiving  Classified  Information/Federal  Express  of  Classified  Material 

An  account  with  the  Base  Information  Transfer  System  (BITS)  office  on  base  was  established  in 
order  to  receive  and  dispatch  official  government  mail.  The  off-base  address  was  used  for  receipt 
of  unclassified  and  nonsensitive  mail.  JADS  used  the  on-base  military  address  to  receive 
incoming  first  class  registered  classified  mail.  Organizations  were  directed  to  use  the  on-base 
address  when  mailing  classified  information  to  JADS. 

Federal  Express  of  Classified  Material.  Classified  packages  were  delivered  to  and  picked  up 
from  our  facility  by  a  Federal  Express  courier.  After  courier  pick-up  hours,  packages  were  hand 
delivered  to  a  Federal  Express  office  location  for  further  delivery. 

A3.  Carrying  Classified  Material  On  and  Off  Base 

A  Department  of  Defense  (DoD)  Form  2501,  Courier  Authorization  Card,  was  issued  to 
individuals  who  had  a  recurrent  need  to  carry  classified  information  to  and  from  the  Air  Force 
base.  The  JADS  administrative  support  team  fell  under  this  provision  during  official  mail  runs. 

A4.  Traveling  Within  the  Continental  United  States  (CONUS)  With  Classified  Information 

Only  JADS  military  members  were  authorized  by  the  joint  test  director  or  security  manager  to 
handcarry  classified  information  to  and  from  the  facility  within  the  CONUS  (DoD  Regulation 
5200.1,  Information  Security  Program).  The  contractor's  home  office  security  manager  made 
arrangements  for  contractors  carrying  classified  material  within  the  CONUS. 

A5.  Visit  Requests 

Visit  requests  for  JADS  military  members  were  dispatched  to  locations  requiring  clearance 
verification.  Their  servicing  home  office  security  manager  initiated  contractor  clearance 
verification  visit  requests.  The  sponsor  unit's  security  manager  approved  need-to-know 
verifications.  A  database  for  incoming  visit  requests  was  established  to  keep  track  of  clearances 
for  intermittent  visits. 
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A6.  Information  Security  Overview 


JADS  employed  current  DoD  security  requirements  and,  when  necessary,  developed  new 
processes  and  procedures  for  designing  a  secure  advanced  distributed  simulation  (ADS) 
architecture.  These  new  processes  and  procedures  are  addressed  in  JADS-level  security 
documentation,  such  as  the  JADS  Security  Management  Plan  and  the  Trusted  Facility  Manual. 
In  addition  to  the  security  concerns  associated  withr  any  military  operation,  such  as  personnel 
security,  there  are  several  security  concerns  associated  with  ADS  testing.  For  example,  security 
instructions  and  regulations  may  vary  at  each  test  site  depending  on  the  classification  of  each 
site’s  network.  Information  on  an  unclassified  network  system  may  be  unclassified  but  sensitive, 
the  network  may  be  classified;  or  JADS  may  require  the  transfer  of  data  from  unclassified 
networks  to  classified  networks.  The  proper  accreditation  and  authorization  at  all  sites  must  be 
obtained.  This  included  establishing  the  accreditation  for  all  automated  information  systems 
(AIS)  equipment  in  JADS  and  establishing  security  memorandums  of  agreement  (MOAs)  with 
other  sites  connected  to  JADS.  Also,  there  were  concerns  associated  with  the  specific  protocols 
used  in  ADS.  For  example,  there  were  distributed  interactive  simulation-specific  concerns  with 
the  broadcasting  of  packets  containing  equipment-specific  data  across  a  network.  Similarly, 
there  were  security  concerns  regarding  the  use  of  multicasting  during  high  level  architecture 
testing. 

A7.  Information  Security  Concerns 

The  test  teams  in  consultation  with  the  security  manager  should  develop  security  requirements. 
The  security  manager  can  not  plan  for  facility  and  information  system  security  until  the  teams 
requirements  are  established. 

A8.  Classified  Processing  Area 

Choose  facilities  that  are  capable  of  supporting  the  level  of  security  that  is  required.  Classified 
processing  areas  may  need  to  be  inspected  for  emissions  security.  The  JADS  classified 
processing  area,  the  Test  Control  and  Analysis  Center  (TCAC),  was  located  in  the  basement  of 
an  old  bank  building  and  was  surrounded  on  two  sides  by  concrete  and  earth.  Its  construction 
and  location  made  it  very  easy  to  convert  into  a  classified  processing  area.  The  only  “beefing  up 
that  was  needed  was  the  placement  of  access  control  devices  and  upgrading  the  electrical  power 
to  support  all  the  data  processing  equipment.  The  physical  location  of  the  classified  processing 
area  was  a  very  important  consideration. 

A9.  Support 

Determine  what  local  tenant  organization  provides  security  support  at  the  location  of  the  test 
sites.  Contact  them  immediately  so  that  a  working  relationship  can  be  established.  Sometimes 
they  may  have  requirements  in  return  for  their  support. 
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AlO.  Service  Agency  Guidelines 

In  the  joint  service  test  arena,  it  is  important  to  determine  which  service  agencies’  security 
regulations  your  organization  will  be  following.  Obtaining  their  regulations  and  manuals  will 
give  you  more  specific  guidelines  in  establishing  a  security  program.  JADS  followed  Air  Force 
and  DoD  guidelines  because  of  the  close  support  received  from  Kirtland  Air  Force  Base  (AFB). 

All.  Multilevel  Processing 

“Multilevel  processing  is  processing  information  with  different  classifications  and  categories  that 
simultaneously  permits  access  by  users  with  different  security  clearances,  but  prevents  users  from 
obtaining  access  to  information  for  which  they  lack  authorization.”  Is  multiple  security  level 
processing  required?  Find  out  the  local  and  service  requirements  to  accomplish  this  mode  of 
processing. 

A12.  Outside  Agency  Security  Contacts 

If  there  are  sites  at  outside  organizations,  contact  them  early  to  establish  MOAs  for  facility, 
networking,  or  computer  system  use  and  placement.  You  will  need  to  establish  and  maintain  a 
close  working  relationship  with  these  people.  After  all,  you  may  be  requesting  to  connect  your 
information  system  into  their  already  established  system.  In  some  cases  this  could  be  a  real 
headache  for  them.  Delays  in  MOAs  can  be  very  time  consuming,  so  your  working  relationship 
with  them  is  critical. 

A 13.  References 

Below  are  some  information  security  references  that  JADS  used.  Depending  on  which  services’ 
guidelines  you  follow,  you  will  either  need  these  or,  at  least,  find  them  very  useful  in  establishing 
information  security  guidelines  and  education. 

•  National  Computer  Security  Center  (NCSC)-TG-005  Version  1,  Trusted  Network 
Interpretation 

•  NCSC-TG-029  Version  1 ,  Introduction  to  Certification  and  Accreditation 

•  Air  Force  Instruction  3 1  -401 ,  Information  Security 

A 14.  Information  Protection 

Information  protection  (IP)  includes  measures  to  protect  friendly  information  systems  by 
preserving  the  availability,  integrity,  and  confidentiality  of  the  systems  and  the  information 
contained  within  the  system.  The  Kirtland  AFB  and  Air  Force  Operational  Test  and  Evaluation 
Center  (AFOTEC)  IP  offices  provided  assistance  in  setting  up  our  security  documents  and 
accreditation  procedures.  All  classified  and  unclassified  information  were  protected  according  to 
the  security  classification  guides,  security  master  plans,  and  program  protection  plans  for  each 
site  and  system.  These  plans  included  the  following  documents: 
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•  Electronic  Security  Command/Joint  Surveillance  System,  Joint  Stars  Security  Classification 
Guide,  Appendix,  9  Mar  94  (This  document  is  classified  secret.) 

•  Navy  Operational  Instruction  C5513.2B,  AIM-9M Security  Classification  Guide,  25  Oct  95 

•  Assistant  Secretary  of  DefenseA^'ME,  Advanced  Medium  Range  Air-to-Air  Missile 
(AMRAAM)  Directorate  of  Engineering,  AMRAAM/HAVE  IMPACT  Security  Classification 
Guide,  1  May  91 

•  Headquarters  United  States  Air  Forces  Europe/TERZ,  HAVE  WINTER  Security  Classification 
Guide,  2  Dec  94 

•  JADS  JTF  Operating  Instruction  33-1,  Security  Requirements  for  Automated  Information 
Systems,  1 8  Jan  96 

If  your  security  personnel  are  new  to  the  field  or  not  experienced  in  the  general  field  of 
information  protection/security,  they  should  contact  the  Defense  Security  Service  (DSS).  The 
DSS  provides  valuable  training  courses. 

•  Department  of  Defense  Regulation  5200-1,  Information  Security  Program 

•  Air  Force  Instruction  33-206.  Information  Protection 

•  http://www.afmc.wpafb.af.mil/HQ-AFMC/SC/cso-scs/scss/compusec.htm 

•  http://www.p-and-e.com/pubs_AF.htm 

•  http :  //Ic  web .  loc .  gov/global/Internet/security  .html 

•  http://www.dss.mil/training/ 

A15.  Computer  Security  (COMPUSEC) 

Computer  security  (also  referred  to  as  automated  information  system  security)  is  an  assembly  of 
computer  hardware,  software  and/or  firmware  configured  to  collect,  create,  communicate, 
compute,  disseminate,  process,  store,  and/or  control  data  or  information.  The  JADS 
COMPUSEC  program  was  operated  under  Air  Force  guidelines.  The  references  listed  below 
were  used  to  establish  that  program. 

•  Air  Force  Systems  Security  Instruction  (AFSSI)  5102,  COMPUSEC  Program 

•  http://lcweb.loc.gov/global/intemet/security.html 

•  http://www.p-and-e.conLL/pubs_AF.htm 

•  Air  Force  Instmction  33-202,  The  Computer  Security  Program 

•  AFSSI  5025,  Volume  I,  The  Certification  and  Accreditation  (C&A)  Process 

•  AFSSI  5024,  Volume  II,  Certifying  Official’s  Handbook 

•  DoD  Computer  Security  Center  (DoD  CSC)-Standard-003-85,  Computer  Security 
Requirements 

•  NCSC-TG-004  Version  1,  Glossary  of  Computer  Security  Terms 

A16.  Emissions  Security 

Emission  security  (EMSEC)  is  the  protection  resulting  from  all  measures  taken  to  deny 
unauthorized  persons  information  of  value  which  might  be  derived  from  intercept  and  analysis  of 
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compromising  emanations  from  crypto-equipment,  information  systems,  and  telecommunications 
systems.  The  JADS  facility  received  communications  services  from  AFOTEC  and  the  base 
information  protection  office.  Because  of  the  heavy  ties  between  JADS  and  the  Air  Force,  all 
AIS  equipment  within  the  JADS  classified  processing  facility  (TCAC)  were  installed  according 
to  Air  Force  emissions  security  requirements.  These  requirements  are  described  in  Air  Force 
Instruction  33-203,  The  Air  Forces  EMSEC  Program,  and  specific  countermeasures  for  JADS 
equipment  are  described  in  Air  Force  Security  Systems  Manual  (AFSSM)  7011,  The  Emission 
Security  Countermeasures  Review.  Emanations  concerns  at  remote  locations  were  addressed  in 
the  MOA  between  JADS  and  the  remote  sites.  The  JADS  designated  approval  authority  and  the 
information  systems  security  officer  ensured  that  the  remote  locations  maintained  emanations 
security  standards  in  accordance  to  those  required  by  the  Air  Force,  even  if  the  remote  location 
was  not  on  an  Air  Force  installation.  An  integral  part  of  the  TCAC  local  area  network  (LAN) 
certification  was  to  have  an  EMSEC  inspection  of  the  facility  that  housed  the  TCAC  LAN.  A 
qualified  inspection  officer  from  AFOTEC  conducted  the  EMSEC  inspection  and  assessment. 

A17.  Communications  Security  (COMSEC) 

COMSEC  includes  measures  taken  to  deny  unauthorized  persons  information  derived  from 
telecommunications  of  the  U.S.  Government  concerning  national  security  and  to  ensure  the 
authenticity  of  such  telecommunications.  Communications  security  includes  crypto-security, 
transmission  security,  emission  security,  and  physical  security  of  communications  security 
material  and  information. 

The  JADS  facility  received  communications  services  from  Kirtland  Air  Force  Base.  COMSEC 
material  was  treated  according  to  the  Secure  Data  Network  System  (SDNS)  User  Representative 
and  COMSEC  Accounting  Procedures,  and  the  JADS  COMSEC  Emergency  Action  Plan.  All 
remote  locations  also  maintained  individual  COMSEC  accounts  and  received  cryptographic 
keying  material  (KEYMAT)  from  the  Air  Force. 

The  base  and  the  Cryptologic  Management  Directorate,  Logistics  Management  Division  of  the 
San  Antonio  Air  Logistics  Center  helped  determine  security  requirements  and  COMSEC 
equipment  capabilities.  After  approval,  JADS  updated  Allowance  Standard  658,  which 
authorizes  COMSEC  equipment;  then,  submitted  an  AF  Form  601,  Equipment  Action  Request,  to 
order  equipment.  The  base  COMSEC  accountant  set  up  a  requirement  for  KEYMAT  and 
equipment.  The  base  COMSEC  office  also  conducted  biannual  COMSEC  inspections. 
Additionally,  JADS  was  required  to  have  safes  for  storage  and  an  emergency  action  plan  to 
ensure  proper  destruction  of  KEYMAT. 
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Annex  B 

Legacy  Issues  and  Lessons  Learned 


BL  Web  Site 

JADS  chose  to  design  and  develop  its  web  site  in  house  using  simple,  static  hypertext  markup 
language  (html)  for  the  majority  of  the  site.  While  html  is  difficult  to  maintain  (it  requires 
manual  update),  it  is  simple  and  straightforward  to  implement.  (If  JADS  was  redesigning  the  site 
today,  a  more  dynamic  approach  using  easily  updated  databases,  Cold  Fusion,  and  style  sheets  in 
place  of  static  html  would  be  considered.)  However,  much  of  the  JADS  web  site  s  success  was 
due  to  the  following  factors,  which  hold  true  no  matter  how  the  site  is  technically  implemented. 

•  Content  is  king.  The  JADS  web  site  was  information  rich,  offering  valuable  insights  into 
distributed  simulation  (what  it  is,  how  it  works),  lessons  learned,  frequently  asked  questions, 
other  distributed  simulation  programs  and  technical  papers  on  every  aspect  of  distributed 
simulation.  The  site  was  designed  to  offer  more  than  a  description  of  the  test.  JADS 
provided  extensive  information,  and  the  test  and  evaluation  and  modeling  and  simulation 
communities  responded  very  positively. 

•  3  clicks  to  content.  Organize  your  information  so  it  is  easy  to  use.  Users  shouldn  t  have  to 
dig  for  information.  By  the  third  mouse  click,  they  should  have  reached  some  value-added 
information.  More  clicks  may  be  required  to  delve  into  detail,  but  the  content  should  be  there 
by  the  third  click. 

•  Presentation  is  critical.  No  matter  how  technical  the  subject  matter,  people  respond  well  to 
attractively  presented  information. 

•  Use  everyday  language  as  much  as  possible,  not  technobabble  or  acronym-ese. 

•  Graphics  count.  Attractive  graphics  add  considerably  to  the  appeal  of  the  site.  Without 
attractive  graphics,  your  site  is  likely  to  get  overlooked.  Graphics  should  be  small  in  file  size, 
however.  People  will  not  wait  long  for  a  page  to  download.  Background  graphics,  which  can 
be  very  small,  can  add  significantly  to  the  page  without  adding  much  to  the  download  time. 
Photographic  images  are  best  compressed  in  Joint  Photographic  Experts  Group  (JPEG) 
format,  while  vector  graphics  are  best  compressed  using  Graphics  Interchange  Format  (GIF) 
format. 

•  Links  can  be  helpful  resources.  JADS  built  an  extensive  list  of  links  related  to  the  subject 
matter.  This  was  beneficial  to  JADS  because  it  brought  customers  back  to  the  site  often  and 
to  the  customer  because  it  provided  a  valuable  service.  However,  limit  outside  links  to  the 
“links”  page  (JADS  called  its  page  “jump  points”)  as  much  as  possible.  Whenever  you 
provide  a  link  to  an  outside  web  site,  youTe  inviting  customers  to  leave  yours,  and  they  may 
never  return. 

•  Use  a  commercial  Internet  service  provider.  A  commercial  Internet  service  provider  can 
be  more  cost  effective,  easier  to  manage,  and  more  reliable  than  maintaining  your  own  server. 
JADS  found  that  it  was  cheaper  to  hire  a  service  provider  (and  prices  keep  going  down); 
someone  who  would  respond  promptly  when  something  went  wrong.  The  service  provider 
guaranteed  high  (99  percent  or  better)  availability.  JADS  originally  hosted  the  site  on  its  own 
server  and  experienced  many  problems  with  base  communications,  and  very  limited, 
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sometimes  long-awaited  support,  and  unpredictable  availability.  The  service  provider  was 
also  a  ready  source  of  expertise  when  needed. 

B2i  Training 

JADS  developed  a  customer  training  program  that  was  very  well  received.  The  course  served  as 
an  excellent  method  to  share  results  and  lessons  learned,  which  was  a  major  objective  of  JADS. 
Customers  were  pleased  with  a  free  training  course,  end  JADS  was  able  to  spread  the  word  to  an 
extensive  group  -  some  1,400  people.  JADS  offered  this  course  locally  each  quarter  and  took  it 
to  many  organizations  that  requested  it.  The  main  secret  to  success  was  getting  the  word  out  via 
E-mail;  web;  attractive,  printed  announcements;  and  actively  following  up  with  customers  who 
expressed  an  interest. 

Customers  seemed  to  like  the  lessons  learned  part  the  best,  especially  JADS’  honesty  and 
willingness  to  share  information. 

JADS  tried  using  interactive  multimedia  as  part  of  the  presentation  but  found  it  cumbersome  for 
the  instructors  when  compared  to  the  familiarity  and  ease  of  using  PowerPoint.  Presentation 
slides  were  enhanced  with  graphics,  which  made  them  quite  large.  So,  presentations  were  split 
into  five  sections.  For  the  course  books,  the  PowerPoint  slides  were  copied  into  Adobe 
Pagemaker  with  a  header  and  footer,  page  numbers,  and  lines  for  notes  on  each  page.  JADS 
added  an  acronym  list,  a  cover  with  a  flashy  graphic,  a  list  of  recommended  resources,  and  table 
of  contents.  Instructors  distributed  and  reviewed  evaluations  after  each  course.  The  course  was 
then  updated  regularly  based  on  comments  or  new  content. 

Part  of  the  training  initiative  was  to  institutionalize  some  of  the  training  products  into  established 
training  programs  and  centers.  JADS  worked  with  Defense  Systems  Management  College, 
among  others,  to  establish  this  remaining  legacy. 

B3.  Multimedia 

The  use  of  multimedia  was  very  helpful  to  JADS.  Interactive,  multimedia  compact  discs  (CDs) 
gave  customers  an  alternative  way  to  see  results  from  JADS  tests  rather  than  reading  extensive 
technical  reports  (which  were  also  available  on  the  CD).  The  CDs  were  extremely  popular  and 
much  easier  to  carry  than  the  reports. 

Multimedia  is  expensive.  Video  is  the  most  expensive  part,  so  choose  carefully  which  pieces  of 
information  to  highlight  with  video.  JADS  spent  about  $50  thousand  to  develop  the  first  two 
CDs  (the  second  was  an  expanded  version  of  the  first)  and  the  first  two  videos.  Much  of  the 
video  development  and  the  digitization  of  the  video  and  narration  were  contracted  out. 

The  subsequent  videos  and  CDs  were  primarily  developed  in-house  and  with  help  from  the 
Kirtland  communications  squadron.  Since  they  were  not  working  on  a  fee-for-service  basis,  the 
cost  of  our  final  three  videos  was  reduced  to  approximately  $5  thousand  apiece.  The  last  four 
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CDs  were  produced  for  between  $2  thousand  and  $4  thousand  apiece  with  most  of  that  cost 
coming  from  our  own  labor. 

JADS  examined  many  different  pieces  of  software  (Macromedia  Authorware,  Macromedia 
Director,  Adobe  Acrobat)  to  implement  the  multimedia  CDs  and  chose  Adobe  Acrobat  because 
of  its  simplicity,  widespread  use,  and  cross-platform  capability. 

JADS  ran  into  some  difficulties  when  trying  to  use  QuickTime  (to  make  the  presentation  cross¬ 
platform)  since  Adobe  Acrobat  3.0  doesn’t  support  QuickTime  3.0.  It  was  decided  to  move  to 
audio  video  interleaved  (AVI)  (video  for  Windows)  and  make  the  CDs  personal  computer  (PC)- 
only. 

Video  clips  were  developed  in  Adobe  Premiere,  an  easy-to-learn  desktop  video  editing  program. 
All  videos  were  scripted  with  a  thorough  review  by  the  JADS  steering  committee  before  editing 
began. 

The  screens  were  laid  out  in  Adobe  Pagemaker,  converted  to  Adobe  Acrobat,  and  linked 
according  to  a  storyboard  approved  by  the  JADS  steering  committee.  Reports  were  separate 
Acrobat  files  linked  from  the  '‘document  library”  portion  of  the  project. 

B4.  Newsletters 

The  JADS  newsletter.  Reality  Check,  was  one  of  the  first  legacy  projects  undertaken.  It  served  a 
vital  role  in  creating  a  customer  base  and  establishing  continuing  communication  between  JADS 
and  its  customers. 

Important  points  about  developing  a  newsletter: 

•  Keep  it  short.  An  11x17  sheet,  double-sided,  folded,  with  an  8.5  x  11  sheet  insert  was  a 
good  length.  This  accommodated  about  7-10  short  articles  per  issue. 

•  Use  color.  Color  is  a  critical  attention-getter. 

•  Make  it  easy  and  enjoyable  to  read.  Use  everyday  language.  Limit  acronyms  and  spell 
them  out  whenever  you  do  use  them.  Use  first  person  as  much  as  possible.  This  makes 
people  feel  like  you’re  talking  to  them. 

•  Offer  it  in  different  formats.  JADS  offered  its  newsletter  on  paper,  on  the  web,  and  via  E- 
mail. 

•  Use  quality  paper.  The  feel  of  quality  makes  a  difference  in  customers’  perception  of  the 
product. 

•  Make  it  easy  to  get.  Customers  could  sign  up  for  the  newsletter  at  conferences  or  on  the 
web.  Also,  every  training  course  attendee  was  added  to  the  newsletter  mailing  list. 

The  JADS  newsletter  was  laid  out  in  Adobe  Pagemaker.  It  was  originally  color  copied  on  base 
until  there  was  a  problem  with  base  graphics  support.  A  cost  comparison— having  it  color  copied 
using  base  printing  versus  buying  color  printers  and  printing  it  in  house— indicated  that  while 
more  labor  intensive,  printing  in  house  was  the  most  cost-effective  option.  Newsletters  were 
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printed  on  inexpensive  Hewlett-Packard  deskjet  1 120C  deskjet  printers.  This  also  gave  JADS  a 
level  of  control  over  its  products  and  production  process  that  was  very  helpful  in  ensuring  its 
products  were  high  quality  and  effective. 

B5.  Brochures 

JADS  needed  an  informational  brochure  from  the  start,  something  to  inform  people  what  JADS 
was  about,  what  it  was  doing,  and  why.  The  legacy  manager  designed  a  color  pocket  folder  that 
was  used  throughout  the  program.  Fact  sheets  enclosed  in  the  pocket  explained  the  three  tests, 
the  analysis  approach,  the  test  control  approach,  and  the  legacy  program  and  products.  The  fact 
sheets  were  updated  to  reflect  current  status. 

Brochure  lessons  learned: 

•  Design  the  front  cover  as  an  attention-getter.  Have  the  main  message  displayed 
prominently.  Don’t  clutter  it  with  information. 

•  Don’t  put  contact  information  and  phone  numbers  on  the  main  printed  brochure.  Use 

stickers  to  attach  that  information  to  the  brochure  itself,  have  a  place  to  insert  a  business  card, 
or  include  the  contact  information  on  the  inserts.  JADS  reorganized  several  times  after  the 
brochure  was  printed,  several  of  the  main  players  left,  and  it  moved  to  a  different  address. 

•  Don’t  try  to  include  too  much  information.  Just  stick  to  the  main  message  on  the  printed 
piece. 

•  Use  attractive  graphics  that  convey  the  subject  matter.  A  signature  graphic  that  the 
organization  uses  in  many  places  (brochure,  booth,  web,  etc.)  is  a  great  way  to  build  identity. 

B6.  Technical  Papers 

One  of  the  ways  JADS  spread  information,  results,  and  lessons  learned  on  various  technical 
aspects  of  our  tests  was  to  present  technical  papers  at  key  forums  that  its  customers  attended. 
Over  60  papers  were  written,  and  delivered  at  more  than  50  conferences.  JADS  targeted  four 
main  conferences  per  year  and  recommended  paper  topics  and  suggested  points  of  contact  for 
each.  Additional  papers  were  always  encouraged,  and  several  were  presented  at  conferences 
other  than  the  four  major  ones.  All  papers  were  hosted  on  the  web  site  after  they  were  presented 
at  the  conference,  and  copies  were  handed  out  at  our  booths.  The  effort  was  well  worth  the  travel 
and  conference  fees.  Without  this  concerted  effort  and  high  level  of  encouragement,  many 
aspects  of  our  tests  would  never  have  been  documented.  Customers  learned  directly  from 
technical  experts  and  had  a  chance  to  critique  and  ask  questions;  JADS  got  a  chance  to  learn 
about  others  who  were  doing  similar  work.  JADS  earned  a  high  level  of  respect  and  credibility 
through  its  people,  and  it  was  good  career  development  for  them  as  well. 

JADS  also  found  key  forums  to  get  involved  in  and  played  a  major  part  in  them.  This  was  done 
to  satisfy  a  part  of  JADS’  charter  -  “Identify  T&E-related  requirements  for  future  ADS  systems”. 
For  example,  JADS  was  involved  in  the  Simulation  Interoperability  Standards  Organization  from 
its  infancy  and  played  a  major  part  in  how  things  were  set  up  and  managed.  JADS  personnel 
volunteered  to  chair  discussion  panels,  conference  sessions,  and  training  courses  at  relevant 
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conferences  and  workshops.  This  effort  gave  JADS  a  level  of  exposure  and  credibility  that  was 
invaluable  to  the  legacy  program  and  the  ability  to  influence  the  direction  of  the  technology  we 
worked  with. 

B7.  Displays/Trade  Shows 

Having  booths  at  relevant  conferences  served  as  the  most  useful  way  to  build  our  mailing  list, 
interface  with  customers  face-to-face,  and  distribute  products. 

Lessons  learned  about  booths: 

•  Use  the  booth  as  an  attention-grabber.  Don’t  try  to  put  too  much  information  on  the  booth 
itself.  Use  a  single  large  graphic  and  a  few  words  to  convey  the  main  message. 

•  Have  informational  handouts.  Handouts  explain  everything  else  you  want  to  say  to  your 
customers,  but  don’t  try  to  cram  too  much  information  into  any  one  handout.  Short  handouts 
are  always  taken,  while  thick  reports  are  generally  passed  by.  Customers  will  often  take 
many,  many  short  handouts,  the  equivalent  of  a  large  report,  but  won’t  take  the  large  reports. 
CDs,  videos  and  diskettes  are  even  more  desirable. 

•  Get  names  and  addresses  of  the  customers  who  visit  the  booth.  Offer  them  something 
(like  a  subscription  to  the  newsletter). 

•  Renting  versus  buying  a  booth.  JADS  decided  to  purchase  a  booth  that  could  be  set  up  a 
number  of  different  ways.  Look  at  your  program  over  its  lifetime  and  hypothesize  different 
ways  you  might  want  to  use  the  booth.  When  choosing  a  booth,  also  take  into  account 
shipping  costs  and  logistics,  as  well  as  storage  space  (or  places  to  hide  your  extra  products). 
Early  in  the  program,  we  covered  the  entire  booth  with  very  large  graphics  to  get  customers 
attention.  Late  in  the  program  we  had  so  many  products  to  distribute,  storage  space  was 
crucial.  The  graphics  had  to  be  smaller,  but  customers  already  knew  us  and  were  looking  for 
information. 

B9.  Videos 

Legacy  struggled  with  the  videos.  While  there  was  a  push  to  develop  videos  early  in  the  program 
(an  introductory  video  would  have  been  helpful),  we  didn’t  have  any  film  to  work  with  or 
anybody  with  video  experience  on  our  team.  When  we  finally  completed  two  videos  several 
years  into  the  project,  they  were  distributed  to  training  organizations  and  customers  unfamiliar 
with  JADS.  Videos  were  also  useful  to  convey  lessons  learned. 

Video  lessons  learned: 

•  Plan  storyboards  well  in  advance  of  the  actual  test  event.  This  allows  you  to  film  what 
you  want  rather  than  “making  do”  with  the  footage  provided. 

•  Gather  photos  and  video  from  events  while  they’re  happening. 

•  Learn  which  outlets  are  okay  for  the  videographers  to  use.  This  may  seem  insignificant, 
but  in  our  first  test,  the  videographer  unknowingly  plugged  his  equipment  into  the  wrong 
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outlet,  blew  a  circuit  and  brought  the  entire  test  control  center  down.  After  that  event,  the 
outlets  were  clearly  marked  and  color  coded. 

•  Don’t  have  lights  on  for  hours  during  a  test.  The  generated  heat  can  shut  down  the 
computers. 

•  You  need  hours  of  footage  for  minutes  of  Hnished  video.  Take  into  account  any  video  you 
might  need  for  multimedia  work  as  well  as  for  videos.  However,  the  more  footage  you  have, 
the  longer  it  will  take  to  select  the  footage  you  want. 

•  Have  someone  from  the  legacy  team  on  site  to  direct  the  videographer.  Make  sure  the 
videographer  shoots  footage  from  several  different  angles  and  under  different  lighting.  You 
will  likely  only  get  one  chance  to  film  a  test  event.  The  people  running  the  test  won’t  want  to 
be  bothered  with  video,  won’t  know  what’s  needed,  and  simply  won’t  put  as  much  effort  into 
it  as  someone  who  is  responsible  for  the  final  product. 

•  Use  stock  footage  as  much  as  possible.  Much  is  available  from  the  Department  of  Defense 
library.  Ask  for  footage  and  photos  from  the  video/communication  squadrons  at  the  bases  or 
posts  involved  in  your  test. 
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Annex  C 

Acronyms  and  Abbreviations 


ACETEF 

Air  Combat  Environment  Test  and  Evaluation  Facility,  Patuxent  River, 
Maryland;  Navy  facility 

ACOT 

Advanced  Communications-Computer  Systems  Officer  Training 

ADEWS 

Advanced  Distributed  Electronic  Warfare  System;  Army  sponsored 

ADS 

advanced  distributed  simulation 

AEC 

Army  Evaluation  Command 

AF 

Air  Force 

AFB 

Air  Force  Base 

AFDTC 

Air  Force  Development  Test  Center,  Eglin  Air  Force  Base,  Florida 

AFEWES 

Air  Force  Electronic  Warfare  Evaluation  Simulator,  Fort  Worth,  Texas; 
Air  Force  managed  with  Lockheed  Martin  Corporation 

AFI 

Air  Force  instruction 

AFMAN 

Air  Force  manual 

AFMC 

Air  Force  Materiel  Command 

AFOTEC 

Air  Force  Operational  Test  and  Evaluation  Center,  Kirtland  Air  Force 
Base,  New  Mexico 

AFPC 

Air  Force  Personnel  Center 

AFSC 

Air  Force  specialty  code 

AFSSI 

Air  Force  systems  security  instruction 

AFSSM 

Air  Force  systems  security  manual 

AIM 

air  intercept  missile 

AIS 

automated  information  systems 

AMG 

Architecture  Management  Group 

AMRAAM 

advanced  medium  range  air-to-air  missile 

AR 

Army  regulation 

ASCAS 

Automated  Security  Clearance  Approval  System 

ASD 

Assistant  Secretary  of  Defense 

ATEC 

Army  Test  and  Evaluation  Command 

AVI 

audio  video  interleaved 

BITS 

Base  Information  Transfer  System 

C3I 

command,  control,  communications  and  intelligence 

C4I 

command,  control,  communications,  computers,  and  intelligence 

C4ISR 

command,  control,  communications,  computers,  intelligence, 
surveillance  and  reconnaissance 

CD 

compact  disk  that  can  hold  a  large  quantity  of  computer  data 

CG 

commanding  general 

COMOPTEVFOR 

Commander,  Operational  Test  and  Evaluation  Force 

COMPUSEC 

computer  security 

COMSEC 

communications  security 

CONUS 

continental  United  States 

CRE 

consolidated  resource  estimate 
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CROSSBOW 

Office  of  the  Secretary  of  Defense  committee  under  the  Director,  Test, 
Systems  Engineering  and  Evaluation 

CSC 

Computer  Security  Center 

DAA 

designated  approval  authority 

DAWIA 

Defense  Acquisition  Workforce  Improvement  Act 

DCSOPS 

deputy  chief  of  staff  for  operations  and  plans 

DDSA 

Deputy  Director,  System  Assessment 

DIS 

distributed  interactive  simulation 

DISA 

Defense  Information  Systems  Agency 

DISN 

Defense  Information  Systems  Network 

DMSO 

Defense  Modeling  and  Simulation  Organization,  Alexandria,  Virginia 

DoD 

Department  of  Defense 

DSMC 

Defense  Systems  Management  College 

DSS 

Defense  Security  Service 

DSSOC 

Defense  Security  Services  Operations  Center 

DT 

developmental  testing 

DT&E 

developmental  test  and  evaluation 

DTC 

Developmental  Test  Command 

DTIC 

Defense  Technical  Information  Center 

E-mail 

electronic  mail 

EMSEC 

emission  security 

ETE 

end-to-end;  JADS  End-To-End  Test 

EW 

electronic  warfare;  JADS  Electronic  Warfare  Test 

FTS 

Federal  Telecommunications  System 

FY 

fiscal  year 

GIF 

Graphics  Interchange  Format 

GPS 

global  positioning  system 

GSA 

General  Services  Administration 

GTRI 

Georgia  Tech  Research  Institute,  Atlanta,  Georgia 

HITE 

hardware-in-the-loop  (electronic  warfare  references) 

HLA 

high  level  architecture 

HQ 

headquarters 

html 

hypertext  markup  language 

HWIL 

hardware-in-the-loop  (system  integration  references) 

ID 

identification 

IMPAC 

International  Merchant  Purchase  Authorization  Card 

IP 

Internet  protocol;  initial  point 

IRIG 

Inter-Range  Instrumentation  Group 

ISSO 

information  systems  security  officer 

ITEA 

International  Test  and  Evaluation  Agency 

JADS 

Joint  Advanced  Distributed  Simulation,  Albuquerque,  New  Mexico 

JFS 

joint  feasibility  study  or  JADS  Joint  Feasibility  Study 

Joint  STARS 

Joint  Surveillance  Target  Attack  Radar  System 

JPEG 

Joint  Photographic  Experts  Group 

JT&E 

joint  test  and  evaluation 
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JTD 

joint  test  director 

JTF 

joint  test  force  or  Joint  Test  Force,  Albuquerque,  New  Mexico 

KEYMAT 

cryptographic  keying  material 

LAN 

local  area  network 

LFP 

live  fly  phase 

M&S 

modeling  and  simulation 

MAJCOM 

major  command 

MIPR 

military  interdepartmental  purchase  request 

MITRE 

a  company  that  provides  engineering  services 

MOA 

memorandum  of  agreement 

MOE 

measure  of  effectiveness 

MOP 

measure  of  performance 

MOS 

military  occupational  specialty 

MOU 

memorandum  of  understanding 

MSIAC 

Modeling  and  Simulation  Information  Analysis  Center 

N&E 

network  and  engineering 

NAG 

National  Assessment  Group 

NAWC-WPNS 

Naval  Air  Warfare  Center  Weapons  Division 

NCO 

noncommissioned  officer 

NCSC 

National  Computer  Security  Center 

O&M 

operations  and  maintenance 

OCA 

original  classification  authority 

ODP 

officer  distribution  plan 

OSD 

Office  of  the  Secretary  of  Defense 

OT 

operational  test 

OT&E 

operational  test  and  evaluation 

OTC 

Operational  Test  Command 

OTP 

outline  test  plan 

PC 

personal  computer 

PERSCOM 

Army  Personnel  Command 

PFMR 

project  funds  management  record 

PGM 

precision  guided  munitions 

PID 

program  introduction  document 

POC 

point  of  contact 

PR 

preliminary  review 

R&D 

research  and  development 

RTI 

runtime  infrastructure 

SAC 

senior  advisory  council 

SAIC 

Science  Applications  International  Corporation 

SAP 

special  access  program 

SCI 

sensitive  compartmented  information 

SCIF 

sensitive  compartmented  information  facility 

SF 

standard  form 

SISO 

Simulation  Interoperability  Standards  Organization 

SIT 

system  integration  test;  JADS  System  Integration  Test 
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srw 

Simulation  Interoperability  Workshop 

soc 

statement  of  capability 

SOP 

standard  operating  procedure 

SPECTRUM® 

an  instrumentation  suite  used  to  measure  bandwidth  utilization 

SPJ 

self-protection  jammer 

sso 

special  security  officer 

T&E 

test  and  evaluation 

TAB 

technical  advisory  board 

TCAC 

Test  Control  and  Analysis  Center,  Albuquerque,  New  Mexico 

TDA 

table  of  distribution  and  allowances 

TDY 

temporary  duty 

TEMPEST 

special  shielding  against  electromagnetic  radiation 

TEXCOM 

U.S.  Army  Test  and  Experimentation  Command 

TRP 

test  resource  plan 

TSARC 

test  schedule  and  review  committee 

TSLA 

Threat  Simulator  Linking  Activity 

USDA&T 

Under  Secretary  of  Defense  for  Acquisition  and  Technology 

VV&A 

verification,  validation,  and  accreditation 

WAN 

wide  area  network 
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